AI RAG Injector

Retrieval-Augmented Generation (RAG) is a technique that improves the accuracy and relevance of language model responses by enriching prompts with external data at runtime. Instead of relying solely on what the model was trained on, RAG retrieves contextually relevant information—such as documents, support articles, or internal knowledge—from connected data sources like vector databases.

This retrieved context is then automatically injected into the prompt before the model generates a response. RAG is a critical safeguard in specialized or high-stakes applications, where factual accuracy matters. LLMs are prone to hallucinations, plausible-sounding but factually incorrect or fabricated responses. RAG helps mitigate this by grounding the model’s output in real, verifiable data.

The following table describes the different use cases for RAG based on industry:

The AI RAG Injector plugin automates the retrieval and injection of contextual data for RAG pipelines without doing manual prompt engineering or retrieval logic. Integrated at the gateway level, it handles embedding generation, vector search, and context injection transparently for each request.

• Simplifies RAG workflows: Automatically embeds prompts, queries the vector DB, and injects relevant context without custom retrieval logic.
• Platform-level control: Shifts RAG logic from app code to infrastructure, allowing platform teams to enforce global policies, update configurations centrally, and reduce developer overhead.
• Improved security: Vector DB access is limited to the AI Gateway, eliminating the need to expose it to individual dev teams or AI agents.
• Enables RAG in restricted environments: Supports RAG even where direct access to the vector database is not possible, such as external-facing or isolated services.
• Developer productivity: Developers can focus on building AI features without needing to manage embeddings, similarity search, or context handling.
• v3.11+ Save LLM costs: When using the AI RAG Injector plugin with the AI Prompt Compressor, you can wrap specific prompt parts in <LLMLINGUA> tags within your template to target only those sections for compression, preserving the rest of the prompt unchanged.

This plugin extends the functionality of the AI Proxy plugin, and requires either AI Proxy or AI Proxy Advanced to be configured first. To set up AI Proxy quickly, see Get started with AI Gateway.

When a user sends a prompt, the RAG Injector plugin queries a configured vector database for relevant context and injects that information into the request before passing it to the language model.

1. You configure the AI RAG Injector plugin via the Admin API or decK, setting up the RAG content to send to the vector database.
2. When a request reaches the AI Gateway, the plugin generates embeddings for request prompts, then queries the vector database for the top-k most similar embeddings.
3. The plugin injects the retrieved content from the vector search result into the request body, and forwards the request to the upstream service.

The following diagram is a simplified overview of how the plugin works. See the following section for a more detailed description.

 
sequenceDiagram
    participant User
    participant AIGateway as AI Gateway (RAG Injector Plugin)
    participant VectorDB as Vector DB (Data Source)
    participant Upstream as Upstream Service

    User->>AIGateway: Send request with prompt
    AIGateway->>VectorDB: Query for similar embeddings
    VectorDB-->>AIGateway: Return relevant context
    AIGateway->>Upstream: Inject context and forward enriched request
    Upstream-->>User: Return response
  

RAG Generation process

The RAG workflow consists of two critical phases:

1. Data preparation: Processes and embeds unstructured data into a vector index for efficient semantic search
2. Retrieval and generation: The system uses similarity search to dynamically assemble contextual prompts that guide the language model’s output.

Phase 1: Data Preparation

This phase sets up the foundation for semantic retrieval by converting raw data into a format that can be indexed and searched efficiently.

Step breakdown:

1. A document loader pulls content from various sources, such as PDFs, websites, emails, or internal systems.
2. The system breaks the unstructured data into smaller, semantically meaningful chunks to support precise retrieval.
3. Each chunk is transformed into a vector embedding (a numeric representation that captures its semantic content).
4. These embeddings are saved to a vector database, enabling a fast, similarity-based search during query time.

Phase 2: Retrieval and Generation

This phase runs in real time, taking user input and producing a context-aware response using the indexed data.

Step breakdown:

1. The user’s query is converted into an embedding using the same model used during data preparation.
2. A semantic similarity search locates the most relevant content chunks in the vector database.
3. The system builds a custom prompt by combining the retrieved chunks with the original query.
4. The LLM generates a contextually accurate response using both the retrieved context and its own internal knowledge.

The diagram below shows how data flows through both phases of the RAG pipeline, from ingestion and embedding to real-time query handling and response generation:

 
sequenceDiagram
    autonumber
    actor User
    participant RawData as Raw Data
    participant EmbeddingModel as Embedding Model
    participant VectorDB as Vector Database
    participant LLM

    par Data Preparation Phase
        activate RawData
        RawData->>EmbeddingModel: Load and chunk documents, generate embeddings
        deactivate RawData

        activate EmbeddingModel
        EmbeddingModel->>VectorDB: Store embeddings
        deactivate EmbeddingModel

        activate VectorDB
        deactivate VectorDB
    end

    par Retrieval & Generation Phase
        activate User
        User->>EmbeddingModel: (1) Submit query and generate query embedding

        activate EmbeddingModel
        EmbeddingModel->>VectorDB: (2) Search vector DB
        deactivate EmbeddingModel

        activate VectorDB
        VectorDB-->>EmbeddingModel: Return relevant chunks
        deactivate VectorDB

        activate EmbeddingModel
        EmbeddingModel->>LLM: (3) Assemble prompt and send
        deactivate EmbeddingModel

        activate LLM
        LLM-->>User: (4) Generate and return response
        deactivate LLM
        deactivate User
    end
  

Rather than guessing from memory, the LLM paired with the RAG pipeline now has the ability to look up the information it needs in real time, which reduces hallucinations and increases the accuracy of the AI output.

A vector database can be used to store vector embeddings, or numerical representations, of data items. For example, a response would be converted to a numerical representation and stored in the vector database so that it can compare new requests against the stored vectors to find relevant cached items.

The AI RAG Injector plugin supports the following vector databases:

• Using config.vectordb.strategy: redis and parameters in config.vectordb.redis:
  • Redis with Vector Similarity Search (VSS)
  • AWS MemoryDB for Redis v3.12+
• Using config.vectordb.strategy: pgvector and parameters in config.vectordb.pgvector:
  • PostgreSQL with pgvector v3.10+

To learn more about vector databases in AI Gateway, see Embedding-based similarity matching in Kong AI gateway plugins.

Once you’ve configured your vector database and ingested content, you can control which Consumers access specific knowledge base articles and refine query results using metadata filters.

Collections

A collection is a logical grouping of knowledge base articles with independent access control rules. When you ingest content via the Admin API, assign it to a collection using the collection field in the metadata.

Example metadata structure:

{
  "content": "Quarterly revenue increased 15%...",
  "metadata": {
    "collection": "finance-reports",
    "date": "2023-10-14",
    "tags": ["finance", "quarterly"],
    "source": "internal"
  }
}

Configuration

Two independent mechanisms control which results consumers receive:

• ACL filtering: Server restricts collections based on Consumer Groups
• Metadata filtering: Clients specify criteria (tags, dates, sources) to narrow results within authorized collections

This configuration creates the following access rules:

• finance-reports: Accessible only to Consumers in the finance or admin groups. Contractors are explicitly denied.
• public-docs: Accessible to all Consumers (empty allow and deny lists).
• Other collections: No access (empty global ACL means deny by default).

plugins:
  - name: ai-rag-injector
    config:
      ...
      consumer_identifier: consumer_group
      global_acl_config:
        allow: []
        deny: []
      collection_acl_config:
        finance-reports:
          allow:
            - finance
            - admin
          deny:
            - contractor
        public-docs:
          allow: []
          deny: []

In this configuration, collections with their own ACL in collection_acl_config ignore global_acl_config entirely. They must explicitly list all allowed subjects.

Check the how-to guide for details about how ACLs work in the AI RAG Injector plugin.

ACL evaluation

The plugin checks access in this order:

1. Deny list: If subject matches, deny access
2. Allow list: If list exists and subject doesn’t match, deny access
3. Empty ACL: If both lists are empty, allow access

Collections with their own ACL in collection_acl_config ignore global_acl_config entirely. They must explicitly list all allowed subjects.

Metadata filtering

LLM clients can refine search results by specifying filter criteria in the query request. Filters apply within the collections. The AI RAG Injector plugin uses a Bedrock-compatible filter grammar with the following operators:

• equals: Exact match
• greaterThan: Greater than (>)
• greaterThanOrEquals: Greater than or equal to (>=)
• lessThan: Less than (<)
• lessThanOrEquals: Less than or equal to (<=)
• in: Match any value in array
• andAll: Combine multiple filter clauses

Review the how-to guide for details about how metadata filtering works.

You can combine multiple conditions with andAll:

{
  "andAll": [
    {"equals": {"key": "source", "value": "internal"}},
    {"in": {"key": "tags", "value": ["finance", "quarterly"]}},
    {"greaterThanOrEquals": {"key": "date", "value": "2023-01-01"}}
  ]
}

Filter parameters:

You can include filters in the ai_rag_injector parameter of your request:

curl "http://localhost:8000/" \
     -H "Content-Type: application/json" \
     --json '{
       "messages": [
         {
           "role": "user",
           "content": "What were Q4 results?"
         }
       ],
       "ai-rag-injector": {
         "filters": {
           "andAll": [
             {
               "equals": {
                 "key": "source",
                 "value": "internal"
               }
             },
             {
               "in": {
                 "key": "tags",
                 "value": [
                   "q4",
                   "quarterly"
                 ]
               }
             }
           ]
         },
         "filter_mode": "strict",
         "stop_on_filter_error": false
       }
     }'

Query flow

The following diagram shows how ACL and metadata filtering work together during query processing:

 
flowchart TB
    Start([Query Request]) --> Auth[Authenticate Consumer]
    Auth --> CheckACL{Authorized
Collections?}
    CheckACL -->|No| Deny[❌ Access Denied]
    CheckACL -->|Yes| HasFilter{Metadata
Filters
Specified?}
    HasFilter -->|No| SearchAll[Search all chunks
in authorized collections]
    HasFilter -->|Yes| FilterMode{filter_mode
setting?}
    FilterMode -->|compatible| SearchCompat[Return chunks matching filter
OR chunks with no metadata]
    FilterMode -->|strict| SearchStrict[Return only chunks
matching filter]
    SearchAll --> Return[✓ Return Results]
    SearchCompat --> Return
    SearchStrict --> Return
  

Admin API

Use the Admin API to ingest content with metadata and collection assignments.

• Ingest chunk:

  POST /ai-rag-injector/{pluginID}/ingest_chunk
  {"content": "...", "metadata": {"collection": "finance-reports", ...}}
  

  Copied!

• Lookup chunks:

  POST /ai-rag-injector/{pluginID}/lookup_chunks
  {"prompt": "...", "collection": "finance-reports", "filters": {...}}
  

  Copied!

  Using cloud authentication with Redis v3.13+

Starting in Kong Gateway 3.13, you can authenticate with a cloud Redis provider for your Redis strategy. This allows you to seamlessly rotate credentials without relying on static passwords.

The following providers are supported:

• AWS ElastiCache
• Azure Managed Redis
• Google Cloud Memorystore (with or without Valkey)

Each provider also supports an instance and cluster configuration.

Important: Kong Gateway open source plugins do not support any Redis cloud provider cluster configurations.

To configure cloud authentication with Redis, add the following parameters to your plugin configuration:

config:
  storage: redis
  storage_config:
    redis:
      host: $INSTANCE_ADDRESS
      username: $INSTANCE_USERNAME
      port: 6379
      cloud_authentication:
        auth_provider: aws
        aws_cache_name: $AWS_CACHE_NAME
        aws_is_serverless: false
        aws_region: $AWS_REGION
        aws_access_key_id: $AWS_ACCESS_KEY_ID
        aws_secret_access_key: $AWS_ACCESS_SECRET_KEY

config:
  storage: redis
  storage_config:
    redis:
      cluster_nodes:
      - ip: $CLUSTER_ADDRESS
        port: 6379
      username: $CLUSTER_USERNAME
      port: 6379
      cloud_authentication:
        auth_provider: aws
        aws_cache_name: $AWS_CACHE_NAME
        aws_is_serverless: false
        aws_region: $AWS_REGION 
        aws_access_key_id: $AWS_ACCESS_KEY_ID
        aws_secret_access_key: $AWS_ACCESS_SECRET_KEY 

config:
  storage: redis
  storage_config:
    redis:
      host: $INSTANCE_ADDRESS
      username: $INSTANCE_USERNAME
      port: 6379
      cloud_authentication:
        auth_provider: azure
        azure_client_id: $AZURE_CLIENT_ID
        azure_client_secret: $AZURE_CLIENT_SECRET
        azure_tenant_id: $AZURE_TENANT_ID

config:
  storage: redis
  storage_config:
    redis:
      cluster_nodes:
      - ip: $CLUSTER_ADDRESS
        port: 6379
      username: $CLUSTER_USERNAME
      port: 6379
      cloud_authentication:
        auth_provider: azure
        azure_client_id: $AZURE_CLIENT_ID
        azure_client_secret: $AZURE_CLIENT_SECRET
        azure_tenant_id: $AZURE_TENANT_ID

config:
  storage: redis
  storage_config:
    redis:
      host: $INSTANCE_ADDRESS
      port: 6379
      cloud_authentication:
        auth_provider: gcp
        gcp_service_account_json: $GCP_SERVICE_ACCOUNT

config:
  storage: redis
  storage_config:
    redis:
      cluster_nodes:
      - ip: $CLUSTER_ADDRESS
        port: 6379 
      port: 6379
      cloud_authentication:
        auth_provider: gcp
        gcp_service_account_json: $GCP_SERVICE_ACCOUNT