AI Prompt Compressor

The Kong AI Prompt Compressor plugin compresses retrieved chunks before sending them to a Large Language Model (LLM), reducing text length while preserving meaning. It uses the LLMLingua 2 library for fast, high-quality compression. The plugin supports:

• Ratio-based or target token compression, for example, reduce a message to 80% of the original length or compress to 150 tokens.
• Configurable compression ranges, for example, compress prompts under 100 tokens with a 0.8 ratio or compress them to exactly 100 tokens.
• Selective compression using <LLMLINGUA>...</LLMLINGUA> tags to target specific sections of the prompt. These tags work only in the inject_template field of the AI RAG Injector plugin and must be used in combination with the AI Prompt Compressor.

Efficient prompt compression helps you manage token limits, cut costs, and speed up LLM requests, all while keeping sensitive data safe and your prompts focused.

The following table outlines common use cases for the plugin and the configuration options available to tailor its behavior.

Kong provides a Docker image for the AI Prompt Compressor service, which compresses LLM prompts before sending them upstream. It uses LLMLingua 2 to reduce prompt size, which helps you manage token limits and maintain context fidelity. The service supports both HTTP and JSON-RPC APIs and is designed to work with the AI Prompt Compressor plugin in AI Gateway.

Kong distributes the AI Prompt Compressor service publicly on Docker Hub, under the kong/ai-compress-service repository. No authentication is required to pull it.

To pull an image:

docker pull kong/ai-compress-service:TAG

Replace TAG with the image tag you want to run, for example:

docker pull kong/ai-compress-service:v0.0.6

You can run the image with:

docker run --rm -p 8080:8080 kong/ai-compress-service:v0.0.6

Check the AI Prompt Compressor documentation for more detail.

Image configuration options

You can configure the Kong Compressor Service using environment variables. These affect model selection, hardware usage, logging, and worker behavior.

Compression endpoints

The compressor service exposes both REST and JSON-RPC endpoints. You can use these interfaces to compress prompts, check the current status, or integrate with upstream services and plugins.

• POST /llm/v1/compressPrompt: Compresses a prompt using either a compression ratio or a target token count. Supports selective compression via <LLMLINGUA> tags.

• GET /status: Returns information about the currently loaded LLMLingua model and device settings (for example, CPU or GPU).

• POST /: JSON-RPC endpoint that supports the llm.v1.compressPrompt method. Use this to invoke compression programmatically over JSON-RPC.

See the AI Prompt Compressor OpenAPI specification for complete details.

The AI Prompt Compressor plugin offers flexible compression controls to fit different use cases. You can choose between full-prompt compression, conditional strategies, or selectively compressing only parts of the prompt:

1. The user sends the final prompt to the AI Prompt Compressor plugin.
2. The plugin checks the prompt for <LLMLINGUA>…</LLMLINGUA> tags.
   • If tags are found, only the tagged sections are sent to LLMLingua 2 for compression.
   • If no tags are found, the entire prompt is sent to LLMLingua 2 for compression.
3. Compression is applied based on configured rules, including ratio, target token count, or conditional length-based rules.
4. The compressed prompt is returned to the plugin.
5. The plugin sends the compressed prompt to the Large Language Model (LLM).
6. The LLM processes the prompt and returns the response to the user.

The following diagram illustrates how the AI Prompt Compressor plugin processes and compresses incoming prompts based on tagging and configured rules.

sequenceDiagram
    actor User
    participant KongAICompressor as AI Prompt Compressor Plugin
    participant LLMLingua2 as LLMLingua 2 Compressor
    participant LLM as Large Language Model

    User->>KongAICompressor: Sends final prompt
    activate KongAICompressor
    KongAICompressor->>KongAICompressor: Check for LLMLINGUA tags

    alt If tagged content found
        KongAICompressor->>LLMLingua2: Compress tagged sections
        activate LLMLingua2
        LLMLingua2-->>KongAICompressor: Return compressed sections
        deactivate LLMLingua2
    else If no LLMlingua tags
        KongAICompressor->>LLMLingua2: Compress entire prompt
        activate LLMLingua2
        LLMLingua2-->>KongAICompressor: Return compressed prompt
        deactivate LLMLingua2
    end

    KongAICompressor->>LLM: Send compressed prompt
    deactivate KongAICompressor
    activate LLM
    LLM-->>User: Return response
    deactivate LLM

The AI Prompt Compressor plugin applies structured compression to preserve essential context of prompts sent by users, rather than trimming prompts arbitrarily or risking token overflows. This ensures the LLM receives a well-formed, focused prompt keeping token usage under control.