All AI Gateway documentation

Overview of AI gateway capabilities

Get started quickly with AI Gateway setup and usage.

Learn about the core capabilities of AI Gateway.

Understand how to manage and govern AI usage effectively.

Explore how AI Gateway helps enforce data governance policies.

Best practices and tools for designing effective prompts.

Implement safeguards to ensure safe and compliant AI outputs.

Customize and transform AI requests with Gateway features.

Learn how AI Proxy streaming works.

Learn about AI Gateway logging capabilities.

Explore how to monitor AI LLM metrics in AI Gateway.

Access advanced analytics features in AI Gateway.

This page is an introduction to the AI providers available in Kong Gateway.

Set up Anthropic chat routing with the AI Proxy Advanced plugin.

Connect Azure AI services using Managed Identity through the AI Proxy Advanced plugin.

Configure Azure chat routing with the AI Proxy Advanced plugin.

Set up Bedrock chat routing with the AI Proxy Advanced plugin.

Integrate Cohere chat routing with the AI Proxy Advanced plugin.

Configure Gemini chat routing with the AI Proxy Advanced plugin.

Set up Hugging Face chat routing with the AI Proxy Advanced plugin.

Integrate Llama 2 chat routing with the AI Proxy Advanced plugin.

Set up Mistral chat routing with the AI Proxy Advanced plugin.

Configure OpenAI chat routing with the AI Proxy Advanced plugin.

Set up Anthropic chat routing with the AI Proxy plugin.

Configure Azure chat routing with the AI Proxy plugin.

Set up Bedrock chat routing with the AI Proxy plugin.

Integrate Cohere chat routing with the AI Proxy plugin.

Configure Gemini chat routing with the AI Proxy plugin.

Set up Hugging Face chat routing with the AI Proxy plugin.

Integrate Llama 2 chat routing with the AI Proxy plugin.

Set up Mistral chat routing with the AI Proxy plugin.

Configure OpenAI chat routing with the AI Proxy plugin.

Use Azure AI Content Safety to check and audit AI Proxy plugin messages before proxying them to an upstream LLM

Prepend or append an array of llm/v1/chat messages to a user's chat history

Check llm/v1/chat or llm/v1/completions requests against a list of allowed or denied expressions

Provide fill-in-the-blank AI prompts to users

The AI Proxy plugin lets you transform and proxy requests to a number of AI providers and models.

The AI Proxy Advanced plugin lets you transform and proxy requests to multiple AI providers and models at the same time. This lets you set up load balancing between targets.

Create RAG pipelines by automatically injecting content from a vector database

Provides rate limiting for the providers used by any AI plugins.

Use an LLM service to transform a client request body prior to proxying the request to the upstream server

Use an LLM service to transform the upstream HTTP(S) prior to forwarding it to the client

Semantically and intelligently create allow and deny lists of topics that can be requested across every LLM.

Protect sensitive information in client request bodies before they reach upstream services

Prepend or append an array of llm/v1/chat messages to a user's chat history

Check llm/v1/chat or llm/v1/completions requests against a list of allowed or denied expressions

Provide fill-in-the-blank AI prompts to users

The AI Proxy plugin lets you transform and proxy requests to a number of AI providers and models.

The AI Proxy Advanced plugin lets you transform and proxy requests to multiple AI providers and models at the same time. This lets you set up load balancing between targets.

Create RAG pipelines by automatically injecting content from a vector database

Provides rate limiting for the providers used by any AI plugins.

Use an LLM service to transform a client request body prior to proxying the request to the upstream server

Use an LLM service to transform the upstream HTTP(S) prior to forwarding it to the client

Semantically and intelligently create allow and deny lists of topics that can be requested across every LLM.

Protect sensitive information in client request bodies before they reach upstream services

Creates a detailed chat history by configuring the plugin to prepend and append multiple messages, allowing any user to continue the conversation.

Prepends a prompt with specific instructions, such as responding in French, to control how the prompt responds.

Configure the AI Prompt Guard plugin with a combination of allow and deny rules.

Configure the AI Prompt Guard plugin to allow messages following a specific pattern.

Configure the AI Prompt Guard plugin to deny messages following a specific pattern.

Configure the AI Prompt Template plugin with a template and block any requests that don't include a template.

Configure the AI Prompt Template plugin with two templates to use for chat routes.

Configure the AI Prompt Template plugin with a template to use for completions routes.

Configure a dynamic route to target multiple Azure OpenAI model deployments.

Configure a chat route that reads the target model from the request path instead of hardcoding it in the configuration.

Configure one route to proxy multiple models deployed in the same Azure instance.

Kong Gateway can attempt to support models that aren’t pre-configured with format transformers or are untested.

Configure a dynamic route to target multiple Azure OpenAI model deployments.

Configure a chat route that reads the target model from the request path instead of hardcoding it in the configuration.

Kong Gateway can attempt to support models that aren’t pre-configured with format transformers or are untested.

Configure the AI RAG Injector plugin to inject content to use Redis as vector database and the OpenAI text-embedding-3-large model for embedding.

Protect your LLM services with rate limiting.

Protect your LLM services by rate limiting requests based on a custom token.

Configure the AI Request Transformer plugin to use OpenAI GPT-4 to add an element to the client request body before proxying it.

Configure the AI Request Transformer plugin to use OpenAI GPT-4 to translate the client request to a different language. In this example, we'll translate the client's question to French.

Configure the AI Response Transformer plugin to use OpenAI GPT-4 to add an element to the response body before returning it to the client.

Configure the AI Response Transfomer plugin to use OpenAI GPT-4 to add a header to the response before returning it to the client.

Allows messages about a topic as long as they don't contain information about another topic.

Only allows messages about a specific topic. For example, only allow messages about cats.

Only denies messages about a specific topic. For example, only deny messages about dogs.

Configure the plugin to use your sanitizer service to anonymize phone numbers in requests before they reach the upstream service.

See the latest updates and improvements for the AI Prompt Decorator plugin.

See the latest updates and improvements for the AI Prompt Guard plugin.

See the latest updates and improvements for the AI Prompt Template plugin.

See the latest updates and improvements for the AI Proxy plugin.

See the latest updates and improvements for the AI Proxy Advanced plugin.

See the latest updates and improvements for the AI RAG Injector plugin.

See the latest updates and improvements for the AI Rate Limiting Advanced plugin.

See the latest updates and improvements for the AI Request Transformer plugin.

See the latest updates and improvements for the AI Response Transformer plugin.

See the latest updates and improvements for the AI Semantic Prompt Guard plugin.

See the latest updates and improvements for the AI Sanitizer plugin.

Set up consistent hashing for load balancing.

Configure load balancing based on the lowest latency.

Set up load balancing based on the lowest usage.

Configure priority-based load balancing.

Set up round-robin load balancing.

Set up semantic load balancing.

Learn how to configure the AI RAG Injector plugin.

Learn how to quickly get started with AI Gateway

Configure the AI Proxy Advanced plugin to create a chat route using Anthropic.

Configure the AI Proxy Advanced plugin to create a chat route using OpenAI.

Configure the AI Proxy plugin to create a chat route using OpenAI.

Learn how to set up Konnect Config Store as a Vault backend and store a Mistral API key.

Learn how to store and rotate secrets in Google Cloud with Kong Gateway, Mistral, and the AI Proxy plugin.

Use the AI Sanitizer plugin to protect sensitive information in requests.

Learn how to use the Azure AI Content Safety plugin.

Connect your LangChain integrations with Kong Gateway with no code changes.

Use a sample Elasticsearch, Logstash, and Kibana stack to visualize data from the AI Proxy plugin.