Related Documentation
Made by
Kong Inc.
Priority
769
Minimum Version
AI Gateway - 2.0
Previous Versions of this page

The AI Prompt Compressor Policy compresses messages before sending them to a Large Language Model (LLM), reducing text length while preserving meaning. It supports multiple cache-aware compression providers.

The following compression providers are available:

  • kong: Use the LLMLingua 2 library to compress prose in user messages.
  • headroom: Use Headroom to compress agent traffic such as tool results, large JSON payloads, search output, and logs returned by tools.

Why use prompt compression

Efficient prompt compression helps you manage token limits, cut costs, and speed up LLM requests, all while keeping sensitive data safe and your prompts focused.

The following table outlines common use cases for the AI Prompt Compressor Policy and the configuration options available to tailor its behavior.

Use case

Description

Cost reduction Reducing token count in prompts decreases API costs when calling large language models, especially for high-volume use cases.
Token limit management Compress verbose inputs like chat history or documents to stay within the LLM’s context window. Prevents truncation of important content.
Latency reduction Smaller prompts result in faster request/response cycles, improving performance for real-time applications like voice assistants.
Dynamic prompt optimization Automatically strip verbose or low-value content before sending to the LLM, keeping the focus on what’s most relevant.

Deterministic compression

Prompt compression is a powerful lever to reduce costs by shrinking the number of input tokens sent to the LLM provider. However, standard compression techniques can alter the text slightly on each run. Because LLM providers offer native prompt caching that relies on exact prefix matches, any variation in the compressed payload triggers a cache miss, forcing you to pay full price for those tokens. Deterministic compression solves this by ensuring that identical inputs always produce identical compressed outputs. This avoids cache misses and guarantees a cache hit on the provider’s side, allowing the AI Gateway to stack the cost-saving benefits of both input token reduction and provider-level caching.

LLMLingua compressor service

Kong provides a Docker image for a compressor service, which compresses LLM prompts before sending them upstream. It uses LLMLingua 2 to reduce prompt size, which helps you manage token limits and maintain context fidelity. The compressor service supports both HTTP and JSON-RPC APIs and is designed to work with the AI Prompt Compressor Policy in AI Gateway.

Access the Docker image

Kong distributes the compressor service publicly on Docker Hub, under the kong/ai-compress-service repository. No authentication is required to pull it.

To pull an image:

docker pull kong/ai-compress-service:TAG

Replace TAG with the image tag you want to run, for example:

docker pull kong/ai-compress-service:v0.0.6

To run the image as a standalone container:

docker run --rm -p 8080:8080 kong/ai-compress-service:v0.0.6

Image configuration options

You can configure the compressor service image using environment variables. These affect model selection, hardware usage, logging, and worker behavior.

Configuration option

Description

LLMLINGUA_MODEL_NAME Specifies the LLMLingua 2 model to use for compression. Defaults to microsoft/llmlingua-2-xlm-roberta-large-meetingbank.
LLMLINGUA_DEVICE_MAP Device on which to run the model. Supported values include cpu, cuda, auto, or mps.
LLMLINGUA_LOG_LEVEL Log level for the LLMLingua compression logic. Set to info, debug, or warning based on your needs.
GUNICORN_WORKERS Number of Gunicorn worker processes (for Docker deployments only). Defaults to 2.
GUNICORN_LOG_LEVEL Log level for Gunicorn server output (for Docker deployments only). Defaults to info.

Compressor Service endpoints

The compressor service exposes both REST and JSON-RPC endpoints. You can use these interfaces to compress prompts, check the current status, or integrate the service with the AI Prompt Compressor Policy and other upstream services.

  • POST /llm/v1/compressPrompt: Compresses a prompt using either a compression ratio or a target token count. Supports selective compression via <LLMLINGUA> tags.

  • GET /status: Returns information about the currently loaded LLMLingua model and device settings (for example, CPU or GPU).

  • POST /: JSON-RPC endpoint that supports the llm.v1.compressPrompt method. Use this to invoke compression programmatically over JSON-RPC.

LLMLINGUA prompt flow

  1. The user sends the final prompt to the AI Prompt Compressor Policy.
  2. The AI Prompt Compressor Policy checks the prompt for <LLMLINGUA>…</LLMLINGUA> tags.
    • If tags are found, only the tagged sections are sent to LLMLingua 2 for compression.
    • If no tags are found, the entire prompt is sent to LLMLingua 2 for compression.
  3. LLMLingua 2 applies the compression using the rule that matches the prompt’s configuration you set with the policy: by ratio, target token count, or conditional length-based rules.
  4. The compressed prompt is returned to the AI Prompt Compressor Policy.
  5. The AI Prompt Compressor Policy sends the compressed prompt to the Large Language Model (LLM).
  6. The LLM processes the prompt and returns the response to the user.

The following diagram illustrates how the AI Prompt Compressor Policy processes and compresses incoming prompts based on tagging and configured rules.

sequenceDiagram
    actor User
    participant KongAICompressor as AI Prompt Compressor Policy
    participant LLMLingua2 as LLMLingua 2 Compressor
    participant LLM as Large Language Model

    User->>KongAICompressor: Sends final prompt
    activate KongAICompressor
    KongAICompressor->>KongAICompressor: Check for LLMLINGUA tags

    alt If tagged content found
        KongAICompressor->>LLMLingua2: Compress tagged sections
        activate LLMLingua2
        LLMLingua2-->>KongAICompressor: Return compressed sections
        deactivate LLMLingua2
    else If no LLMlingua tags
        KongAICompressor->>LLMLingua2: Compress entire prompt
        activate LLMLingua2
        LLMLingua2-->>KongAICompressor: Return compressed prompt
        deactivate LLMLingua2
    end

    KongAICompressor->>LLM: Send compressed prompt
    deactivate KongAICompressor
    activate LLM
    LLM-->>User: Return response
    deactivate LLM

The AI Prompt Compressor Policy applies structured compression to preserve essential context of prompts sent by users, rather than trimming prompts arbitrarily or risking token overflows. This ensures the LLM receives a well-formed, focused prompt keeping token usage under control.

Prompt compression options

The AI Prompt Compressor Policy offers flexible compression controls to fit different use cases. You can choose between full-prompt compression, conditional strategies, or selectively compressing only parts of the prompt:

Configuration Option

Description

Compression by ratio Compress the prompt to a percentage of its original length (for example, reduce to 80%). This allows for consistent shrinkage regardless of the initial size.
Compression by token count Compress the prompt to a specific token target (for example, 150 tokens). Useful when working close to LLM context window limits.
Conditional rules Apply different compression strategies based on prompt length. For example, compress prompts under 100 tokens using a 0.8 ratio, and compress longer prompts to a fixed token count.
Selective compression with tags Wrap sections of the prompt in <LLMLINGUA>...</LLMLINGUA> to target only specific parts for compression, preserving untagged content as-is.

Headroom compressor service

Before you use Headroom with the AI Prompt Compressor Policy, you need a Headroom instance accessible to your AI Gateway.

You can do this with one of the following:

Configure Headroom connection

To configure an AI Prompt Compressor Policy with Headroom as the compressor service, do the following:

Set timeout higher than the sum of two Headroom waits. The first is the per-session turn lock, which makes concurrent turns of one conversation wait for each other (10 seconds). The second is the compression work itself, which HEADROOM_COMPRESSION_TIMEOUT_SECONDS limits (30 seconds by default). The default timeout of 10000 milliseconds is lower than that sum, so requests can time out before Headroom finishes.

For more details, see the configuration reference.

Compressor Service endpoint

The compressor service exposes a /v1/compress endpoint that compresses messages and returns them. This endpoint accepts OpenAI and Anthropic’s message formats. Requests in unsupported formats are forwarded unchanged. You can use this interface to compress prompts or integrate the service with the AI Prompt Compressor Policy.

Headroom uses loopback-trust by default. It answers unauthenticated calls on 127.0.0.1, and returns 404 to non-loopback callers unless it’s started with HEADROOM_COMPRESS_ALLOW_REMOTE=1. You can run Headroom as a co-located sidecar reachable from the AI Gateway data plane without authentication. Alternatively, you can point the Policy at a remote instance and configure a bearer token, sent as both the X-Headroom-Proxy-Token and Authorization: Bearer headers, set with the headroom.proxy_token configuration field. Configuration validation requires headroom.proxy_token unless compressor_url points to a loopback address (localhost, 127.0.0.0/8, or ::1).

Headroom prompt flow

The AI Prompt Compressor Policy uses Headroom in a stateful mode and derives a session identifier for each conversation. Headroom can recognize the turns it has already compressed for that conversation. Recognized turns are replayed unchanged instead of compressed again. This ensures the upstream LLM provider’s cache hits on repeated turns.

The Policy derives the session identifier as follows:

  • If any header listed in config.headroom.session_id_headers is present on the request, the identifier is built from all of the present headers together. Configuring several headers narrows the session instead of replacing it.
  • If none of those headers is present, the identifier is built from the model, the leading system message, and the first user message.

The model name is always part of the identifier. Every identifier is also scoped to the Policy instance and to the AI Consumer, or to the credential when the route authenticates without resolving a consumer.

We recommend headers whose values stay the same across turns, because a header that changes on every request stops the upstream cache from hitting. A header that identifies the caller instead of the conversation, such as an installation or machine ID, places all of that caller’s conversations in one session.

Two requests share one session state on the Headroom side when they have the same Policy instance, the same AI Consumer or credential, and the same model. They must also have the same values for the configured headers or, when no header is present, the same system message and first user message. If sessions collide, then only one will hit the upstream cache.

  1. AI Gateway sends the user or agent’s request to the AI Prompt Compressor.
  2. The AI Prompt Compressor takes the request body, which contains the whole conversation in a messages or input array.
  3. The AI Prompt Compressor sends a POST request to Headroom’s /v1/compress endpoint with the request body, the model name, and a session identifier for the conversation. The model that AI Gateway selects for the route takes precedence over the model in the request body.
  4. If Headroom recognizes the session identifier, it replays the turns it already compressed unchanged and compresses only the new messages. Otherwise, it compresses the whole conversation.
  5. Headroom returns 200 with metadata and the rewritten request body, which includes system and tools.

If the call to Headroom fails, the AI Prompt Compressor rejects the request by default. For details, see Failure behavior.

The following diagram illustrates how the AI Prompt Compressor Policy processes and compresses incoming prompts using Headroom:

sequenceDiagram
    actor User as User/Agent
    participant KongAICompressor as AI Prompt Compressor Policy
    participant Headroom
    participant LLM as LLM Provider

    User->>KongAICompressor: Sends request
    activate KongAICompressor
    KongAICompressor->>Headroom: POST /v1/compress with request body and session ID

    alt Session recognized
        Headroom->>Headroom: Replay cached turns, compress only new messages
    else New session
        Headroom->>Headroom: Compress whole conversation
    end

    alt Compression succeeds
        Headroom-->>KongAICompressor: Return 200 with rewritten request body and metadata
        KongAICompressor->>LLM: Send compressed request to upstream provider
        LLM-->>KongAICompressor: Return response
        KongAICompressor-->>User: Return response
    else Compression fails
        Headroom-->>KongAICompressor: Return failure response
        KongAICompressor--xUser: Return HTTP 500, request not forwarded
    end
    deactivate KongAICompressor

Failure handling

By default, if the call to Headroom fails, times out, or returns an error, the Policy returns an HTTP 500 to the client rather than forwarding the request uncompressed.

The Policy retries a 503 response that indicates a compression timeout, for up to three attempts in total, with a wait of 0.2 seconds and then 0.5 seconds between attempts. Headroom can return this response when a concurrent turn of the same conversation holds the per-session turn lock. After the last attempt, the Policy returns an HTTP 500. Every other non-200 response (for example a 400 from a bad configuration, or a 401 from a missing or incorrect bearer token) fails immediately.

A call to Headroom that reuses a pooled connection Headroom has already closed is sent again on a new connection instead of failing the request.

Limitations

  • Deterministic compression requires sessions which is only available in Headroom v0.37.0 or newer. Older images silently ignore the session id and run stateless.
  • By default, Headroom keeps sessions in memory in one Headroom process, and Kong must send all turns of a session to that process. Point each data plane node at its own sidecar, or point all data plane nodes at one shared instance. Do not put a load balancer in front of Headroom instances that keep sessions in memory.
  • To run several Headroom instances behind a load balancer, set HEADROOM_COMPRESS_SESSION_STORE to a Redis URL (redis://...) on every instance. The instances then share the session state. Shared sessions require Headroom v0.42.0 or later, installed with the redis extra. The default Headroom image doesn’t include the redis extra, so build an image that installs headroom-ai[proxy,redis]. The lock that keeps the turns of one session in order works only inside one instance. If a client sends turns of one session in parallel, route each session to one instance. For more information, see the Headroom repository.
  • No MCP tool-response compression: only the LLM request path is supported.
  • Only requests with a messages or input conversation array are sent to Headroom. Other formats are forwarded uncompressed.
  • A native Bedrock Converse request is skipped, with a warning. Headroom can’t read Converse’s content blocks.
  • The Policy forwards a request uncompressed, regardless of stop_on_error, in the following cases:
    • The request body isn’t JSON.
    • No model name is found in the request or in the model that AI Gateway selects for the route.
  • Compressing a Responses API request (input) requires a Headroom image with input support.

Prompt compression metrics

When the AI Prompt Compressor Policy compresses a request, AI Gateway reports the compression result to Konnect Analytics. Requests that don’t pass through the Policy don’t change any compression metric. You can use these metrics to measure how many tokens compression saves and how long it takes.

Compression metrics are reported with the existing AI Gateway dimensions, so you can scope them by AI Gateway, control plane, data plane node, route, and AI Consumer. The Policy configuration doesn’t change, and the compression log entry (compress_items) keeps its fields and values.

Compression metrics contain token and message counts only. They never include the original or compressed text.

Analytics metrics

The following metrics are available in Explorer:

Metric

Type

Description

save_token_count Count Number of tokens saved, summed over the compressed messages. This metric is reported per compression provider and has no model dimension.
original_token_count Count Number of tokens before compression.
compress_token_count Count Number of tokens after compression.
compressed_message_count Count Number of messages sent to the compressor per request, including messages it leaves unchanged.
duration Latency Average time the compressor took, in milliseconds.

Analytics dimensions

You can also group and filter the compression metrics by these compression-specific dimensions:

Dimension

Values

Description

provider kong, headroom The compression provider that handled the request.
compress_type rate, target_token The compression method used. Available for the kong provider only. For headroom, the value is empty.

The two providers report compression at different granularity. The kong provider reports one result for each compressed message. Headroom reports one result for the whole request. The metrics sum the results for each request, so they mean the same thing for both providers.

Log fields

The summary is also added to the ai.compressor entry of the log serializer, beside compress_items. Logging Policies such as HTTP Log and File Log include these fields:

Field

Description

provider The compression provider, kong or headroom.
original_token_count Number of tokens before compression.
compress_token_count Number of tokens after compression.
save_token_count Number of tokens saved.
compressed_message_count Number of compressed messages.
duration Time the compressor took, in milliseconds.
compress_type The compression method. Not set for headroom.

Headroom session fields

For the headroom provider, the single compress_items entry has an information string in this format:

<transforms>;frozen_message_count=<N>;cached_prefix_replayed=<true|false>

Headroom adds the two session parts only when it runs the request in a session.

Part

Description

<transforms> A comma-separated list of the transforms that Headroom applied to the request.
frozen_message_count The number of leading messages that Headroom did not compress again on this turn. Headroom counts the leading messages that it can replay from the compression cache of the session. Plain messages always count. A tool result counts only after Headroom caches it. The last message never counts. Headroom calculates this value from its own state. The value can be more than zero on the first turn.
cached_prefix_replayed true when Headroom compressed some earlier messages to different bytes than on the previous turn, and then replaced them with the bytes from the previous turn. The replacement keeps the start of the prompt the same. false when Headroom replaced nothing. This is the usual value. The value is also false when the Headroom process has no previous turn for the session:
  • The first turn of a session.
  • After the session state expires.
  • When Headroom keeps sessions in memory, on an instance that hasn’t served the session before.

In these cases, the start of the prompt can change.

Do not use cached_prefix_replayed to measure prompt cache hits. The field reports whether Headroom replayed earlier bytes. It does not report on the prompt cache of the LLM provider. A value of false does not mean that the start of the prompt is the same as on the previous turn. To measure cache hits, use the cached-token counts that the LLM provider reports. These are cached_tokens in ai.proxy.usage.prompt_tokens_details in the audit log, and the gen_ai.usage.cache_read.input_tokens attribute in OpenTelemetry traces.

Response headers

When the compressor handles a request, the Policy adds the following headers to plain and streamed responses, so the caller can read what compression saved:

Header

Value

X-Kong-Prompt-Compression-Saved-Tokens Number of tokens saved.
X-Kong-Prompt-Compression-Messages Number of compressed messages.
X-Kong-Prompt-Compression-Duration Time the compressor took, in milliseconds.

The headers are also added when the compressor doesn’t change a message because its token count falls outside every compression range. In that case, the message still counts toward the message count and the original token count, and it saves no tokens.

If the compressor returns an empty result, or the request body has a shape the provider can’t compress, the Policy forwards the request to the upstream unchanged, regardless of stop_on_error. In that case, the response has no compression headers, and the request adds nothing to the compression metrics.

Forward proxy support

Set config.proxy on this entity to route its outbound requests through an HTTP forward proxy. Use this in network-isolated deployments where AI Gateway cannot open direct connections to LLM providers or auxiliary services.

The proxy record is identical for AI Model, AI MCP Server, and supported AI Policy entities. Existing capabilities such as load balancing, health checking, streaming, WebSocket, and HTTP/2 continue to work when the proxy is active.

For the full field reference, traffic flow, and limitations, see Forward proxy support.

Looking for on-prem configuration? Convert the kongctl configuration to a decK yaml file and use deck file ai2kong. See Configure Kong AI Gateway on-prem for more information.

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!