Related Documentation
Made by
Kong Inc.
Priority
2500
Minimum Version
AI Gateway - 2.0
Tags

Crawlers and scripted clients can call your AI Models, AI Agents, and AI MCP Servers and use up your provider tokens and upstream capacity. The Bot Detection AI Policy blocks these requests in AI Gateway based on the User-Agent header, before they reach the upstream. It blocks a built-in list of common bots, and you can add your own patterns to deny or allow specific clients.

How does the Bot Detection Policy work?

The Bot Detection Policy compares the User-Agent header of each request against the built-in list and your custom patterns. When the header matches the built-in list or a deny pattern and no allow pattern matches, AI Gateway rejects the request with a 403 Forbidden response.

AI Gateway sends the 403 response before it forwards the request upstream, so a blocked streaming request doesn’t open a stream. Requests without a User-Agent header pass through.

The AI Auth Strategy runs before the Bot Detection Policy. When an AI Model requires authentication, a bot without valid credentials receives a 401 response from the AI Auth Strategy before the Bot Detection Policy runs.

Built-in bot list

The Bot Detection Policy blocks common bots by default, such as the Googlebot and bingbot search engine crawlers.

The built-in list also includes HTTP client libraries that scripts use. If your applications, SDKs, or agent frameworks use one of these libraries, the Policy blocks them unless you allow them:

  • Go-http-client
  • axios
  • okhttp
  • Python-urllib
  • Apache-HttpClient
  • reqwest
  • aws-sdk-* and Boto3

For a configuration that uses the built-in list, see Block common bots.

Deny or allow custom User-Agents

The Bot Detection Policy’s config.deny setting blocks additional clients, and config.allow lets specific clients through. Both settings take a list of regular expressions that the Policy matches against any part of the User-Agent header.

An allow pattern takes precedence over the built-in list and over deny. When a User-Agent matches both an allow pattern and a deny pattern, AI Gateway accepts the request.

Patterns are case-sensitive. To match a pattern regardless of case, start it with (?i).

For a configuration that allows a Go HTTP client and blocks scrapers, see Allow and deny custom User-Agents.

Looking for on-prem configuration? Convert the kongctl configuration to a decK yaml file and use deck file ai2kong. See Configure Kong AI Gateway on-prem for more information.

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!