AI Proxy Advanced

The AI Proxy Advanced plugin lets you transform and proxy requests to multiple AI providers and models at the same time. This lets you set up load balancing between targets.

AI Proxy Advanced plugin accepts requests in one of a few defined and standardized OpenAI formats, translates them to the configured target format, and then transforms the response back into a standard format.

v3.10+ To use AI Proxy Advanced with non-OpenAI format without conversion, see section below for more details.

AI Proxy Advanced plugin supports capabilities across batch processing, multimodal embeddings, agents, audio, image, streaming, and more, spanning multiple providers.

For Kong Gateway versions 3.6 or earlier:

• Chat Completions APIs: Multi-turn conversations with system/user/assistant roles.

• Completions API: Generates free-form text from a prompt.

  OpenAI has marked this endpoint as legacy and recommends using the Chat Completions API for developing new applications.

See the following table for capabilities supported in AI Gateway:

The following providers are supported by the legacy Completions API:

• OpenAI
• Azure OpenAI
• Cohere
• Llama2
• Amazon Bedrock
• Gemini
• Hugging Face

AI Gateway supports proxying requests to the following AI providers. Each provider page documents supported capabilities, configuration requirements, and provider-specific details.

For detailed capability support, configuration requirements, and provider-specific limitations, see the individual provider reference pages.

The AI Proxy Advanced plugin will mediate the following for you:

• Request and response formats appropriate for the configured config.targets[].model.provider and config.targets[].route_type
• The following service request coordinates (unless the model is self-hosted):
  • Protocol
  • Host name
  • Port
  • Path
  • HTTP method
• Authentication on behalf of the Kong API consumer
• Decorating the request with parameters from the config.targets.model[].options block, appropriate for the chosen provider
• Recording of usage statistics of the configured LLM provider and model into your selected Kong log plugin output
• Optionally, additionally recording all post-transformation request and response messages from users, to and from the configured LLM
• Fulfillment of requests to self-hosted models, based on select supported format transformations

Flattening all of the provider formats allows you to standardize the manipulation of the data before and after transmission. It also allows your to provide a choice of LLMs to the Kong Gateway Consumers, using consistent request and response formats, regardless of the backend provider or model.

v3.11+ AI Proxy Advanced supports REST-based full-text responses, including RESTful endpoints such as llm/v1/responses, llm/v1/files, llm/v1/assisstants and llm/v1/batches. RESTful endpoints support CRUD operations— you can POST to create a response, GET to retrieve it, or DELETE to remove it.

AI Gateway transforms requests and responses according to the configured config.targets[].model.provider and config.targets[].route_type, using the OpenAI format by default. v3.10+ To use a provider’s native format instead, set config.llm_format to a value other than openai. The plugin then passes requests upstream without transformation. See Supported native LLM formats for available options.

The following table maps each route type to its OpenAI API reference and generative AI category. See the AI provider reference pages for provider-specific details.

Provider-specific parameters can be passed using the extra_body field in your request. See the sample OpenAPI specification for detailed format examples.

If you use a provider’s native SDK, AI Gateway v3.10+ can proxy the request and return the upstream response without payload format conversion. Set config.llm_format to a value other than openai to preserve the provider’s native request and response formats.

In this mode, AI Gateway will still provide analytics, logging, and cost calculation. When config.llm_format is set to a native format, only the corresponding provider is supported with its specific APIs.

AI Proxy Advanced supports several load balancing algorithms for distributing requests across AI models:

• Round-robin: Weighted traffic distribution.
• Consistent-hashing: Sticky sessions based on header values.
• Least-connections: Route to backends with spare capacity.
• Lowest-latency: Route to fastest-responding models.
• Lowest-usage: Route based on token counts or cost.
• Semantic: Route based on prompt-to-model similarity.
• Priority: Tiered failover across model groups.

For detailed algorithm descriptions and selection guidance, see Load balancing algorithms.

For load balancing across Gateway Upstreams and Targets instead of LLMs, see load balancing with Kong Gateway.

The AI load balancer supports configurable retries, timeouts, and failover to different models when a target is unavailable.

v3.10+ Fallback works across targets with any supported format. You can mix providers freely, for example OpenAI and Mistral. Earlier versions require compatible formats between fallback targets. For configuration details, see Retry and fallback configuration.

Client errors don’t trigger failover. To failover on additional error types, set config.balancer.failover_criteria to include HTTP codes like http_429 or http_502, and non_idempotent for POST requests.

The AI load balancer supports circuit breakers to improve reliability. If a target reaches the failure threshold defined by config.balancer.max_fails, the load balancer stops routing requests to it until the timeout period (config.balancer.fail_timeout) elapses.

For configuration details and behavior examples, see Circuit breaker.

The plugin allows you to substitute values in the config.targets[].model.name and any parameter under config.targets.model[].options with specific placeholders, similar to those in the Request Transformer Advanced plugin.

The following templated parameters are available:

• $(headers.header_name): The value of a specific request header.
• $(uri_captures.path_parameter_name): The value of a captured URI path parameter.
• $(query_params.query_parameter_name): The value of a query string parameter.

You can combine these parameters with an OpenAI-compatible SDK in multiple ways using the AI Proxy and AI Proxy Advanced plugins, depending on your specific use case:

A vector database can be used to store vector embeddings, or numerical representations, of data items. For example, a response would be converted to a numerical representation and stored in the vector database so that it can compare new requests against the stored vectors to find relevant cached items.

The AI Proxy Advanced plugin supports the following vector databases:

• Using config.vectordb.strategy: redis and parameters in config.vectordb.redis:
  • Redis with Vector Similarity Search (VSS)
  • AWS MemoryDB for Redis v3.12+
• Using config.vectordb.strategy: pgvector and parameters in config.vectordb.pgvector:
  • PostgreSQL with pgvector v3.10+

To learn more about vector databases in AI Gateway, see Embedding-based similarity matching in Kong AI gateway plugins.

Using cloud authentication with Redis v3.13+

If your plugin uses a Redis datastore, you can authenticate to it with a cloud Redis provider. This allows you to seamlessly rotate credentials without relying on static passwords.

The following providers are supported:

• AWS ElastiCache
• Azure Managed Redis
• Google Cloud Memorystore (with or without Valkey)

config:
  vectordb:
    strategy: redis
    redis:
      host: $INSTANCE_ADDRESS
      username: $INSTANCE_USERNAME
      port: 6379
      cloud_authentication:
        auth_provider: aws
        aws_cache_name: $AWS_CACHE_NAME
        aws_is_serverless: false
        aws_region: $AWS_REGION
        aws_access_key_id: $AWS_ACCESS_KEY_ID
        aws_secret_access_key: $AWS_ACCESS_SECRET_KEY

config:
  vectordb:
    strategy: redis
    redis:
      cluster_nodes:
      - ip: $CLUSTER_ADDRESS
        port: 6379
      username: $CLUSTER_USERNAME
      port: 6379
      cloud_authentication:
        auth_provider: aws
        aws_cache_name: $AWS_CACHE_NAME
        aws_is_serverless: false
        aws_region: $AWS_REGION 
        aws_access_key_id: $AWS_ACCESS_KEY_ID
        aws_secret_access_key: $AWS_ACCESS_SECRET_KEY 

config:
  vectordb:
    strategy: redis
    redis:
      host: $INSTANCE_ADDRESS
      username: $INSTANCE_USERNAME
      port: 6379
      cloud_authentication:
        auth_provider: azure
        azure_client_id: $AZURE_CLIENT_ID
        azure_client_secret: $AZURE_CLIENT_SECRET
        azure_tenant_id: $AZURE_TENANT_ID

config:
  vectordb:
    strategy: redis
    redis:
      cluster_nodes:
      - ip: $CLUSTER_ADDRESS
        port: 6379
      username: $CLUSTER_USERNAME
      port: 6379
      cloud_authentication:
        auth_provider: azure
        azure_client_id: $AZURE_CLIENT_ID
        azure_client_secret: $AZURE_CLIENT_SECRET
        azure_tenant_id: $AZURE_TENANT_ID

config:
  vectordb:
    strategy: redis
    redis:
      host: $INSTANCE_ADDRESS
      port: 6379
      cloud_authentication:
        auth_provider: gcp
        gcp_service_account_json: $GCP_SERVICE_ACCOUNT

config:
  vectordb:
    strategy: redis
    redis:
      cluster_nodes:
      - ip: $CLUSTER_ADDRESS
        port: 6379 
      port: 6379
      cloud_authentication:
        auth_provider: gcp
        gcp_service_account_json: $GCP_SERVICE_ACCOUNT