AI Semantic Prompt Guard Policy

The AI Semantic Prompt Guard Policy enforces prompt governance using semantic similarity matching. It compares incoming requests against your configured allow and deny lists, preventing misuse of text completion requests.

You can use a combination of allow and deny rules to maintain integrity and compliance when serving an LLM service using AI Gateway.

How it works

By default, the Policy only evaluates the latest message from the user role. To also match earlier turns in a multi-turn conversation, set config.rules.match_all_conversation_history to true. To also match messages from roles other than user, set config.rules.match_all_roles to true.

The matching behavior is as follows:

  • If any deny prompts are set and the request matches a prompt in the deny list, the request is blocked.
  • If any allow prompts are set, but the request matches none of the allowed prompts, the request is also blocked.
  • If any allow prompts are set and the request matches one of the allow prompts, the request passes through to the LLM.
  • If there are both deny and allow prompts set, the deny condition takes precedence over allow. Any request that matches a prompt in the deny list is blocked, even if it also matches a prompt in the allow list. If the request doesn’t match a prompt in the deny list, then it must match a prompt in the allow list to be passed through to the LLM.

Rejection modes

The AI Semantic Prompt Guard Policy responds to a blocked request according to its rejection mode. In the default none mode, the response is a 400 with the message bad request, which hides that a guardrail is in use.

Use config.rejection_mode to control how AI Gateway responds to a blocked request:

Mode

Status code

Response body

none (default) 400 A JSON error with the block message, for example {"error": {"message": "..."}}. The response omits the guardrail’s name. Clients can tell it apart from a malformed request only by its message, because the status code is the same.
stealth 403 A generic JSON error, {"error": {"message": "request forbidden"}}. The response hides all guardrail details.
verbose 403 A structured JSON error that names the Policy that blocked the request and the reason. See Verbose response.

Use none or stealth for clients outside your organization, so responses don’t reveal how your guardrails are configured. Use verbose in development or for trusted internal clients, to see why a request was blocked.

Verbose response

In verbose mode, the response always has the status code 403 and the error code GUARDRAIL_BLOCKED. Clients can check the error.code field to tell a guardrail rejection apart from other errors:

  • type: Always guardrail_rejected.
  • plugin: The type of the Policy that blocked the request.
  • reason: Why the request was blocked.
  • code: Always GUARDRAIL_BLOCKED.
  • detail: Additional context about the match. This field is only present when the Policy has details to report.

The AI Semantic Prompt Guard Policy evaluates the request before it reaches the LLM, so a blocked request returns the same error response whether or not the request sets stream to true.

For example, a request that matches a deny prompt returns the following response in verbose mode. The detail field contains the prompt that matched and its score, which is the distance between the request and that prompt (a lower score means a closer match):

{
  "error": {
    "type": "guardrail_rejected",
    "plugin": "ai-semantic-prompt-guard",
    "reason": "prompt pattern is blocked",
    "code": "GUARDRAIL_BLOCKED",
    "detail": {
      "matched_prompt": "How do I pick a lock?",
      "score": 0.12
    }
  }
}

To return this response, set config.rejection_mode to verbose:

Detect without blocking

To run the AI Semantic Prompt Guard Policy without blocking, set config.continue_on_detection to true. When the Policy detects a match, it lets the request or response continue. The caller receives the normal response, so config.rejection_mode has no effect while this setting is enabled.

Detections are still recorded in the AI Gateway logs. The ai.proxy.guardrail_triggered object is populated with the Policy that matched and the direction of the match. To include the matched content, also set config.log_blocked_content to true. For the full list of fields, see the AI Gateway audit log reference.

This is useful when you roll out new rules: run them in detection-only mode, review the logs for false positives, then set continue_on_detection back to false to enforce them.

Policy scope

Don’t apply the AI Semantic Prompt Guard Policy to the AI Model that serves its own embedding requests.

The Policy calls an embedding model to compute semantic similarity. By default, it sends that request directly to the embedding provider. If you set config.embeddings.model.options.upstream_url to an AI Model on the same AI Gateway instance, the embedding request goes through the data plane like any other request. When the Policy is global, it also applies to that request, which triggers another embedding request, and so on. The loop doesn’t end, and every request returns an HTTP 500 error.

Important: When the AI Semantic Prompt Guard Policy is global and upstream_url points back to the same AI Gateway instance, every request fails with an HTTP 500 error because of a circular embedding loop.

Use one of the following options to avoid the loop:

  • Attach the Policy to the AI Models it protects. Create it with global: false and reference it in each AI Model’s policies array. The embedding request then goes to an AI Model that the Policy doesn’t cover. See AI Policy scopes.
  • Send embedding requests directly to the embedding provider. Set upstream_url to the provider’s endpoint, or leave it unset. The request bypasses AI Gateway routing, so you must protect the embedding endpoint at the infrastructure level.

Vector databases

A vector database stores and compares vector embeddings—numerical representations of text, prompts, documents, or other content. When you configure semantic features in AI Models or AI Policies, embeddings are generated and stored in the vector database so that incoming requests can be compared against the stored vectors to find semantically similar matches. For example, an incoming prompt is embedded and compared against cached prompt keys, model descriptions, document chunks, or allow/deny lists to determine semantic similarity.

AI Gateway semantic features support the following vector databases:

  • Using vectordb.strategy: redis and parameters in vectordb.redis:
    • Redis with Redis Vector Search
    • Redis Cloud
    • Valkey: When you configure vectordb.strategy: redis, Kong Gateway queries the server and checks the server name field. If it detects Valkey request, it automatically uses the Valkey-specific driver.
    • Managed Redis with cloud authentication:
      • AWS ElastiCache (auth_provider: aws)
      • Azure Managed Redis (auth_provider: azure)
      • Google Cloud Memorystore (auth_provider: gcp)

      For configuration details, see Using cloud authentication with Redis.

  • Using vectordb.strategy: pgvector and parameters in vectordb.pgvector:

Configure vector database settings in AI Models and AI Policies to enable semantic similarity features.

Redis version and module requirements

Kong’s semantic features use the Redis Query Engine (RediSearch) to create and search vector indexes, and the Redis JSON data type to store indexed documents. Your Redis or Valkey deployment must support both, or vector index creation fails.

  • Self-hosted Redis: Use Redis Open Source 8.0 or later, which bundles the Redis Query Engine and JSON data type by default, or a distribution that adds them as modules, such as Redis Stack, Redis Enterprise, or Redis Cloud.
  • Self-hosted Valkey: Use Valkey 8.1 or later with the valkey-search and valkey-json modules enabled. A vanilla Valkey build without these modules can’t be used for semantic features.
  • AWS ElastiCache: ElastiCache doesn’t support custom Redis modules, so ElastiCache for Redis OSS can’t be used for semantic features at any version. Use ElastiCache for Valkey 8.2 or later, which has vector search built in natively.
  • Azure Managed Redis: Enable the RediSearch and RedisJSON modules when you create the instance. Modules can only be selected at creation time and can’t be added afterward.
  • Google Cloud Memorystore: Use Memorystore for Redis Cluster 8.0 or later or Memorystore for Valkey 8.0 or later, both of which include the required modules automatically. Standard (non-cluster) Memorystore for Redis instances, including the Basic tier, are frozen at Redis 7.2 and don’t support RediSearch or the JSON data type, so they can’t be used for semantic features.

Using cloud authentication with Redis

If your AI Policy uses a Redis datastore, you can authenticate to it with a cloud Redis provider. This allows you to rotate credentials without relying on static passwords.

The following providers are supported:

  • AWS ElastiCache
  • Azure Managed Redis
  • Google Cloud Memorystore (with or without Valkey)

Each provider also supports an instance and cluster configuration.

Forward proxy support

Set config.proxy on this entity to route its outbound requests through an HTTP forward proxy. Use this in network-isolated deployments where AI Gateway cannot open direct connections to LLM providers or auxiliary services.

The proxy record is identical for AI Model, AI MCP Server, and supported AI Policy entities. Existing capabilities such as load balancing, health checking, streaming, WebSocket, and HTTP/2 continue to work when the proxy is active.

For the full field reference, traffic flow, and limitations, see Forward proxy support.

Known limitations

Denied-topic matching compares the embedding model similarity score of a prompt to the deny prompts, and scores for the same prompt can differ by language. A prompt in another language can fall below the similarity threshold and bypass a deny prompt. Test the embedding model with the languages your users send. Combine the AI Semantic Prompt Guard Policy with other controls to block content.

Looking for on-prem configuration? Convert the kongctl configuration to a decK yaml file and use deck file ai2kong. See Configure Kong AI Gateway on-prem for more information.

FAQs

Yes, the AI Semantic Prompt Guard Policy supports multilingual input. Support depends on the capabilities of the configured embedding model. The AI Policy sends raw UTF-8 text to an embedding provider supported by AI Gateway (for example, Azure, Bedrock, Gemini, Hugging Face, Mistral, or OpenAI). Similarity scores for the same prompt can differ by language, so test the embedding model with the languages your users send. See Known limitations.

If you see the following error in the logs:

failed to create memorydb instance failed to create index: LIMIT Number of indexes (11) exceeds the limit (10)

This means that the hardcoded MemoryDB instance limit has been reached. To resolve this, create more MemoryDB instances to handle multiple AI Semantic Prompt Guard policy instances.

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!