Dashboards that show traffic and latency per entity help you find slow upstreams, error spikes, and unexpected usage before users report them.
The Datadog Policy is an AI Policy that sends request metrics from AI Gateway, such as request count, request and response size, and latency, to a Datadog agent.
The policy sends the metrics over UDP in the DogStatsD format, so the agent must be reachable from the data plane at the address set in config.host and config.port.
Datadog Policy
Metrics
The Datadog Policy sends the following metrics.
Each metric name starts with the config.prefix value, which is kong by default:
|
Metric |
Name in Datadog |
Description |
Default type |
|---|---|---|---|
request_count
|
kong.request.count
|
Counts the request. |
counter
|
request_size
|
kong.request.size
|
Tracks the request body size in bytes. |
timer
|
response_size
|
kong.response.size
|
Tracks the response body size in bytes. |
timer
|
latency
|
kong.latency
|
Tracks the time AI Gateway took to handle the request, including the response from the upstream. |
timer
|
upstream_latency
|
kong.upstream_latency
|
Tracks the time the upstream took to process the request.
A value of -1 means AI Gateway didn’t forward the request to the upstream.
Filter out -1 values when you use upstream_latency in latency calculations.
|
timer
|
kong_latency
|
kong.kong_latency
|
Tracks the time AI Gateway spent processing the request itself, separate from the upstream. |
timer
|
When you don’t set config.metrics, the policy sends all six metrics with the default types and adds the tag app:kong.
When you set config.metrics, the policy sends only the listed metrics and doesn’t add app:kong.
Metric types
The stat_type field of each metric in config.metrics sets the metric type.
The policy sends each type to the agent as a DogStatsD type code, for example c for counter and ms for timer.
Metrics of type counter or gauge require a sample_rate.
AI Gateway rejects the policy when a counter or gauge metric doesn’t set one.
Tags
The policy adds the following tags to each metric:
name: The name of the AI Agent or AI MCP Server that handled the request. To change the tag name, setconfig.service_name_tag. For AI Models, see Limitations.status: The HTTP status code of the response. To change the tag name, setconfig.status_tag.app:kong: The policy adds this tag only when you don’t setconfig.metrics.
You can add more tags:
- Set
tagson a metric to add your own tags, for exampleenv:production. - Set
config.route_name_tagto add a tag with the name of the route that AI Gateway generated for the entity. - Set
consumer_identifieron a metric to add the caller as a tag. The default tag name isconsumer, and you can change it withconfig.consumer_tag.
Identify AI Consumers
The consumer_identifier field of a metric sets which value of the AI Consumer the policy adds:
consumer_id: The ID of the AI Consumer, with hyphens replaced by underscores. For a configuration, see Log request count.username: The name of the AI Consumer. For a request that authentication rejects, the tag value isanonymous.custom_id: AI Consumers don’t have a custom ID, so the policy doesn’t add a tag. Useconsumer_idorusernameinstead.
Which requests the policy reports
The Datadog Policy reports every request sent to the entity it’s attached to, including requests that end early.
For example, the policy reports a request that authentication rejects with the 401 status code.
The policy reports a request that another policy ends, such as the Request Termination Policy, with that policy’s status code and an upstream_latency of -1.
For streaming responses, latency covers the whole streamed response.
Scopes
You can apply the Datadog Policy globally, or attach it to an AI Model, an AI Agent, an AI MCP Server, or an AI Consumer. For details about how scopes work, see AI Policy scopes.
- At AI Agent and AI MCP Server scope, the policy reports requests to that entity.
- At AI Model scope, the policy reports requests to that AI Model.
See Limitations for the
nametag. - At AI Consumer scope, the policy reports only requests authenticated as that AI Consumer.
-
At global scope, the policy reports requests to every entity on the gateway. To report only some traffic, set a condition on the policy. For example, the following condition reports only requests that include the header
x-rt: on:condition: 'http.headers.x_rt == "on"'
Use AI Consumer scope to count requests per caller, and use global scope when every entity needs the same metrics.
Queuing
The Datadog Policy uses internal queues to decouple the production of log entries from their transmission to the upstream server.
With queuing, entries are placed in a configurable queue before being sent in batches to the upstream server. This has the following benefits:
- Reduces concurrency on the upstream server
- Helps deal with temporary outages of the upstream server due to network or administrative changes
- Can reduce resource usage both in AI Gateway and on the upstream server by collecting multiple entries from the queue in one request
Note: Because queues are structural elements for components in AI Gateway, they only live in the main memory of each worker process and aren’t shared between workers. Therefore, queued content isn’t preserved under abnormal operational situations, like power loss or unexpected worker process shutdown due to memory shortage or program errors.
You can configure several parameters for queuing:
|
Parameters |
Description |
|---|---|
Queue capacity limits:
config.queue.max_entries
config.queue.max_bytes
config.queue.max_batch_size
|
Configure sizes for various aspects of the queue: maximum number of entries, batch size, and queue size in bytes.
When a queue reaches the maximum number of entries and another entry is enqueued, the oldest entry in the queue is deleted to make space for the new entry. The queue code logs a warning when it reaches a capacity threshold of 80% and when it starts to delete entries from the queue, and logs again when the situation normalizes. |
Timer usage:
config.queue.concurrency_limit
|
Only one timer is used to start queue processing in the background by default. Once the queue is empty, the timer handler terminates, and a new timer is created as soon as a new entry is pushed onto the queue. |
Retry logic:
config.queue.initial_retry_delay
config.queue.max_coalescing_delay
config.queue.max_retry_delay
config.queue.max_retry_time
|
If a queue fails to process, it can automatically retry if the failure is temporary, for example due to network problems or upstream unavailability. |
A larger config.queue.max_batch_size makes the policy wait for a full batch before it sends.
A single request’s metrics are sent to the agent up to config.queue.max_coalescing_delay seconds late.
Keep these values low for near-real-time dashboards.
Limitations
At AI Model scope, the Datadog Policy sets the name tag of every metric to ai-gateway, regardless of which AI Model handled the request.
The route_name_tag tag doesn’t identify the AI Model either, because it contains a generated route name.
As a result, you can’t tell AI Models apart in Datadog by grouping on name.
To tell AI Models apart, create one Datadog Policy per AI Model, attach each policy to its AI Model, and add a tag that identifies the AI Model to each metric in config.metrics, for example model:gpt-4o.
Use this workaround only with a policy attached to a single AI Model.
Looking for on-prem configuration? Convert the
kongctlconfiguration to a decKyamlfile and usedeck file ai2kong. See Configure Kong AI Gateway on-prem for more information.