Related Documentation
Made by
Kong Inc.
Incompatible with
on-prem
Priority
100
Minimum Version
AI Gateway - 2.0

Repeating the same request to a slow, rate-limited, or per-request-billed upstream wastes time and money. The Proxy Cache AI Policy stores upstream responses in memory and serves repeated requests from the cache. AI Gateway serves the cached response until the entry expires.

The policy caches a response when the request method, the response status code, and the response content type all match the policy configuration.

The Proxy Cache Advanced Policy extends the Proxy Cache Policy with Redis, Redis Cluster, and Redis Sentinel support.

Use cases

The following are common reasons to cache responses:

Use case

Description

An AI Agent sends the same GET request many times Cache successful GET responses for a fixed time, so repeated requests don’t reach a slow or rate-limited upstream.
Responses differ by tenant or page Add the headers or query parameters that change the response to the cache key, so each variation gets its own entry.
Clients must be able to request fresh data Turn on config.cache_control so a client Cache-Control request header can skip the cache.
Only some clients need caching Attach the policy to an AI Consumer or AI Consumer Group instead of the AI Agent or an AI MCP Server.
You need to know where a response came from Read the X-Cache-Status response header to see whether AI Gateway served the response from the cache.

How does the Proxy Cache Policy work?

When a request matches the policy, AI Gateway builds a cache key and looks it up in memory. If an entry exists, AI Gateway returns it without calling the upstream. If not, AI Gateway sends the request upstream and stores the response when it meets the policy configuration.

The policy stores entries for the number of seconds set in config.cache_ttl. After an entry expires, the next request goes to the upstream and the response replaces the entry.

What does the policy cache?

By default, the policy caches responses that meet all of the following conditions:

The content type must match one of the listed values exactly. To cache a streamed response, add text/event-stream to config.content_type.

Requests to AI MCP Servers use POST. To cache them, add POST to config.request_method. Because the cache key excludes the request body, add POST only for endpoints where the body doesn’t change the response. See Limitations.

Cache key

AI Gateway builds the cache key from the following parts of the request:

Use these settings to control which requests share a cache entry:

  • By default, all query parameters are part of the key. Set config.vary_query_params to use only the listed parameters.
  • By default, the path is case-sensitive. Set config.ignore_uri_case to true to treat /Items and /items as the same path.
  • By default, no request headers are part of the key. List the headers that change the response in config.vary_headers.

AI Gateway returns the key of each request in the X-Cache-Key response header.

Cache status

AI Gateway reports the cache result in the X-Cache-Status response header:

  • Miss: The request matches the policy configuration, but no entry exists. AI Gateway sent the request upstream and stored the response.
  • Hit: AI Gateway served the response from the cache. The age header shows how many seconds ago the entry was stored.
  • Bypass: The policy didn’t use the cache, for example because the request method, response status code, or response content type isn’t in the policy configuration.

To remove a header from responses, set X-Cache-Key, X-Cache-Status, or age in config.response_headers to false.

Scopes

You can apply the Proxy Cache Policy globally, or attach it to an AI Model, an AI Agent, an AI MCP Server, an AI Consumer, or an AI Consumer Group. For details about how scopes work, see AI Policy scopes.

Attach the policy to an AI Agent or an AI MCP Server to cache the traffic of one entity. Attach it to an AI Consumer or an AI Consumer Group when only some clients need caching.

  • When it’s scoped to an AI Agent or AI MCP Server, the Policy caches responses for requests to that entity. Use this scope to cache the traffic of one entity.
  • When it’s scoped to an AI Consumer or AI Consumer Group, the Policy caches responses for authenticated requests from that AI Consumer or from members of that AI Consumer Group. Use this scope when only some clients need caching.
  • When the Policy is scoped to an AI Model, AI Gateway sends every request to the upstream. See Limitations.

Execution order

The Proxy Cache Policy has a priority of 100. For details, see AI Policy priority.

Authentication and Policies with a higher priority run before the cache. For example, an unauthenticated request receives a 401 response, and the IP Restriction Policy rejects a denied client even when a cached entry exists.

A Request Termination Policy has a lower priority, so it runs after the cache. A request that hits the cache returns the cached response even when it carries the trigger header of the Request Termination Policy.

Limitations

  • The policy stores entries in memory only. The only supported value for config.strategy is memory.
  • To clear an entry, wait for config.cache_ttl to expire. The AI Gateway API doesn’t have an endpoint to look up, delete, or purge cache entries.
  • When it’s scoped to an AI Model, the policy sets X-Cache-Status: Bypass on every response and AI Gateway sends every request to the upstream, so AI Gateway doesn’t cache AI Model responses. This applies to chat completion requests, with and without streaming.
  • The cache key is built without the request body. At AI Agent and AI MCP Server scope, POST requests to the same path with different bodies share one cache entry, so the cached response answers all of them until it expires.

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!