Crawlers and scripted clients can call your AI Models, AI Agents, and AI MCP Servers and use up your provider tokens and upstream capacity.
The Bot Detection AI Policy blocks these requests in AI Gateway based on the User-Agent header, before they reach the upstream.
It blocks a built-in list of common bots, and you can add your own patterns to deny or allow specific clients.
Bot Detection Policy
How does the Bot Detection Policy work?
The Bot Detection Policy compares the User-Agent header of each request against the built-in list and your custom patterns.
When the header matches the built-in list or a deny pattern and no allow pattern matches, AI Gateway rejects the request with a 403 Forbidden response.
AI Gateway sends the 403 response before it forwards the request upstream, so a blocked streaming request doesn’t open a stream.
Requests without a User-Agent header pass through.
The AI Auth Strategy runs before the Bot Detection Policy.
When an AI Model requires authentication, a bot without valid credentials receives a 401 response from the AI Auth Strategy before the Bot Detection Policy runs.
Built-in bot list
The Bot Detection Policy blocks common bots by default, such as the Googlebot and bingbot search engine crawlers.
The built-in list also includes HTTP client libraries that scripts use. If your applications, SDKs, or agent frameworks use one of these libraries, the Policy blocks them unless you allow them:
Go-http-clientaxiosokhttpPython-urllibApache-HttpClientreqwestaws-sdk-*andBoto3
For a configuration that uses the built-in list, see Block common bots.
Deny or allow custom User-Agents
The Bot Detection Policy’s config.deny setting blocks additional clients, and config.allow lets specific clients through.
Both settings take a list of regular expressions that the Policy matches against any part of the User-Agent header.
An allow pattern takes precedence over the built-in list and over deny.
When a User-Agent matches both an allow pattern and a deny pattern, AI Gateway accepts the request.
Patterns are case-sensitive.
To match a pattern regardless of case, start it with (?i).
For a configuration that allows a Go HTTP client and blocks scrapers, see Allow and deny custom User-Agents.
Looking for on-prem configuration? Convert the
kongctlconfiguration to a decKyamlfile and usedeck file ai2kong. See Configure Kong AI Gateway on-prem for more information.