Proxy Caching Advanced Policy

Repeating the same request to a slow, rate-limited, or per-request-billed upstream wastes time and money. The Proxy Cache Advanced AI Policy stores upstream responses in memory or in Redis and serves repeated requests from the cache. AI Gateway serves the cached response until the entry expires.

This policy extends the Proxy Cache Policy with Redis storage.

Use cases

The following are common reasons to cache responses:

Use case

Description

An AI Agent sends the same GET request many times Cache successful GET responses for a fixed time, so repeated requests don’t reach a slow or rate-limited upstream.
You want cache entries outside the data plane Set config.strategy to redis to store entries in Redis instead of memory.
A Redis outage must not break requests Set config.bypass_on_err to true so AI Gateway sends the request upstream when Redis fails.
Responses differ by tenant or page Add the headers or query parameters that change the response to the cache key, so each variation gets its own entry.
Only some clients need caching Attach the policy to an AI Consumer or AI Consumer Group instead of the AI Agent or an AI MCP Server.
You need to know where a response came from Read the X-Cache-Status response header to see whether AI Gateway served the response from the cache.

How does the Proxy Cache Advanced Policy work?

The Proxy Cache Advanced Policy builds a cache key for each request that matches the policy configuration and looks the key up in the storage backend. If an entry exists, AI Gateway returns it without calling the upstream. If not, AI Gateway sends the request upstream and stores the response when it meets the policy configuration.

The policy stores entries for the number of seconds set in config.cache_ttl. After an entry expires, the next request goes to the upstream and the response replaces the entry.

What does the policy cache?

The Proxy Cache Advanced Policy caches a response when the request method, the response status code, and the response content type all match the policy configuration. Set these conditions in config.request_method, config.response_code, and config.content_type.

The response content type must match one of the config.content_type values exactly. For a configuration that caches successful GET responses, see Cache responses to successful GET requests.

Storage strategies

The config.strategy setting chooses where AI Gateway stores cache entries:

  • memory: AI Gateway stores entries in the memory of the data plane node.
  • redis: AI Gateway stores each entry as a hash in a Redis database. Set the connection in config.redis, for example config.redis.host and config.redis.port. The Redis entry expires after config.cache_ttl seconds, or after config.storage_ttl seconds when you set it. For a Redis configuration, see Cache responses with Redis.

If Redis fails, for example when the database is down, the result depends on config.bypass_on_err:

Storage TTL

The config.storage_ttl setting keeps entries in the Redis database for longer than config.cache_ttl. When an entry is older than config.cache_ttl and config.cache_control is false, the next request goes to the upstream. AI Gateway replaces the entry with the response and sets the X-Cache-Status header to Refresh.

Cache key

AI Gateway builds the cache key from the following parts of the request:

AI Gateway returns the key of each request in the X-Cache-Key response header.

Cache status

AI Gateway reports the cache result in the X-Cache-Status response header:

  • Miss: The request matches the policy configuration, but an entry doesn’t exist. AI Gateway sent the request upstream and stored the response.
  • Hit: AI Gateway served the response from the cache. The age header shows how many seconds ago the entry was stored.
  • Refresh: The entry was older than config.cache_ttl. AI Gateway sent the request upstream and replaced the entry.
  • Bypass: The policy didn’t use the cache, for example because the request method, response status code, or response content type isn’t in the policy configuration.

Scopes

You can apply the Proxy Cache Advanced Policy globally, or attach it to an AI Model, an AI Agent, an AI MCP Server, an AI Consumer, or an AI Consumer Group. For details about how scopes work, see AI Policy scopes.

Global scope applies the policy to all traffic, so use it only when every entity needs the same caching.

  • When it’s scoped to an AI Agent or AI MCP Server, the Policy caches responses for requests to that entity. Use this scope to cache the traffic of one entity.
  • When it’s scoped to an AI Consumer or AI Consumer Group, the Policy caches responses for authenticated requests from that AI Consumer or from members of that AI Consumer Group. Use this scope when only some clients need caching.
  • When the Policy is scoped to an AI Model, AI Gateway sends every request to the upstream. See Limitations.

Execution order

The Proxy Cache Advanced Policy has a priority of 100. For details, see AI Policy priority.

Authentication and Policies with a higher priority run before the cache. For example, an unauthenticated request receives a 401 response, and the IP Restriction Policy rejects a denied client even when a cached entry exists.

A Request Termination Policy has a lower priority, so it runs after the cache. A request that hits the cache returns the cached response even when it carries the trigger header of the Request Termination Policy.

Limitations

  • AI Gateway doesn’t have an endpoint to look up, delete, or purge cache entries. To clear an entry, wait for config.cache_ttl, or config.storage_ttl when you set it, to expire.
  • When the policy is scoped to an AI Model, the policy sets X-Cache-Status: Bypass on every response and AI Gateway sends every request to the upstream, so AI Model responses are never cached.

Looking for on-prem configuration? Convert the kongctl configuration to a decK yaml file and use deck file ai2kong. See Configure Kong AI Gateway on-prem for more information.

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!