Announcing AI Gateway

[subnetmarco]

Hello everyone 👋 , today we announced a set of new AI plugins in the broader Kong Gateway 3.6 release. You can read the full AI announcement here.

Essentially we can now use Kong Gateway to proxy AI traffic to cloud and self-hosted LLMs with a whole set of L7 AI Capabilities when it comes to prompts, observability and request/response transformation, and more. All other plugins can now be also used on top of upstream AI services to establish more advanced governance, security and traffic control patterns.

You can learn more about the new AI capabilities and even see demonstration videos at the official product page.

I am opening this discussion thread to collect feedback and ideas around this new AI use-case, and looking for feedback and suggestions from the community.

[kandakji]

Hi Marco @subnetmarco , this is great. We'd be happy to contribute to this AI proxy to add support for Amazon Bedrock invoke model API in order to add it to the list of providers you support

[subnetmarco]

That would be awesome to have.

[tysoekong]

Hi @kandakji , I built the initial range of provider integrations to support this AI category.

Would really like to meet, so we can plan out adding support for the Bedrock Inference format! We want to enable as many providers and model variety as possible.

[subnetmarco]

Update: AWS Bedrock will be released in the upcoming v3.8 this summer, along with other LLM technologies.

[subnetmarco]

A new release of AI Gateway has been announced with streaming support, new plugins, richer analytics and more: https://konghq.com/blog/product-releases/ai-gateway-goes-ga

We are going to release AWS Bedrock and Gemini support in just a few weeks.

The new analytics allows to extract all sort of metrics from your GenAI usage:

[guleng]

@subnetmarco I have many LLM models for ai proxy, can I directly load balance and perform health checks on multiple models? I was unable to achieve load balancing for the LLM model in my actual Kong configuration

[subnetmarco]

We are going to be announcing new load balancing capabilities soon that will allow to:

• Load balance across multiple combinations of providers/models.
• Choose between 6 different load balancing algorithms:
  • Round robin
  • Weighted
  • Least busy (least number of ongoing calls)
  • Lowest usage (lowest TPM usage)
  • Lowest latency
  • Semantic Routing
• Implement failovers and retries.

Among many other features that we have been building for the past few months.

[guleng]

Do you have an approximate time?

[subnetmarco]

August/Sept.