Related Documentation
Made by
Kong Inc.
Priority
10
Minimum Version
AI Gateway - 2.0

Dashboards that show traffic and latency per entity help you find slow upstreams, error spikes, and unexpected usage before users report them. The Datadog Policy is an AI Policy that sends request metrics from AI Gateway, such as request count, request and response size, and latency, to a Datadog agent. The policy sends the metrics over UDP in the DogStatsD format, so the agent must be reachable from the data plane at the address set in config.host and config.port.

Metrics

The Datadog Policy sends the following metrics. Each metric name starts with the config.prefix value, which is kong by default:

Metric

Name in Datadog

Description

Default type

request_count kong.request.count Counts the request. counter
request_size kong.request.size Tracks the request body size in bytes. timer
response_size kong.response.size Tracks the response body size in bytes. timer
latency kong.latency Tracks the time AI Gateway took to handle the request, including the response from the upstream. timer
upstream_latency kong.upstream_latency Tracks the time the upstream took to process the request.

A value of -1 means AI Gateway didn’t forward the request to the upstream. Filter out -1 values when you use upstream_latency in latency calculations.
timer
kong_latency kong.kong_latency Tracks the time AI Gateway spent processing the request itself, separate from the upstream. timer

When you don’t set config.metrics, the policy sends all six metrics with the default types and adds the tag app:kong. When you set config.metrics, the policy sends only the listed metrics and doesn’t add app:kong.

Metric types

The stat_type field of each metric in config.metrics sets the metric type. The policy sends each type to the agent as a DogStatsD type code, for example c for counter and ms for timer.

Metrics of type counter or gauge require a sample_rate. AI Gateway rejects the policy when a counter or gauge metric doesn’t set one.

Tags

The policy adds the following tags to each metric:

You can add more tags:

  • Set tags on a metric to add your own tags, for example env:production.
  • Set config.route_name_tag to add a tag with the name of the route that AI Gateway generated for the entity.
  • Set consumer_identifier on a metric to add the caller as a tag. The default tag name is consumer, and you can change it with config.consumer_tag.

Identify AI Consumers

The consumer_identifier field of a metric sets which value of the AI Consumer the policy adds:

  • consumer_id: The ID of the AI Consumer, with hyphens replaced by underscores. For a configuration, see Log request count.
  • username: The name of the AI Consumer. For a request that authentication rejects, the tag value is anonymous.
  • custom_id: AI Consumers don’t have a custom ID, so the policy doesn’t add a tag. Use consumer_id or username instead.

Which requests the policy reports

The Datadog Policy reports every request sent to the entity it’s attached to, including requests that end early. For example, the policy reports a request that authentication rejects with the 401 status code. The policy reports a request that another policy ends, such as the Request Termination Policy, with that policy’s status code and an upstream_latency of -1.

For streaming responses, latency covers the whole streamed response.

Scopes

You can apply the Datadog Policy globally, or attach it to an AI Model, an AI Agent, an AI MCP Server, or an AI Consumer. For details about how scopes work, see AI Policy scopes.

  • At AI Agent and AI MCP Server scope, the policy reports requests to that entity.
  • At AI Model scope, the policy reports requests to that AI Model. See Limitations for the name tag.
  • At AI Consumer scope, the policy reports only requests authenticated as that AI Consumer.
  • At global scope, the policy reports requests to every entity on the gateway. To report only some traffic, set a condition on the policy. For example, the following condition reports only requests that include the header x-rt: on:

    condition: 'http.headers.x_rt == "on"'

Use AI Consumer scope to count requests per caller, and use global scope when every entity needs the same metrics.

Queuing

The Datadog Policy uses internal queues to decouple the production of log entries from their transmission to the upstream server.

With queuing, entries are placed in a configurable queue before being sent in batches to the upstream server. This has the following benefits:

  • Reduces concurrency on the upstream server
  • Helps deal with temporary outages of the upstream server due to network or administrative changes
  • Can reduce resource usage both in AI Gateway and on the upstream server by collecting multiple entries from the queue in one request

Note: Because queues are structural elements for components in AI Gateway, they only live in the main memory of each worker process and aren’t shared between workers. Therefore, queued content isn’t preserved under abnormal operational situations, like power loss or unexpected worker process shutdown due to memory shortage or program errors.

You can configure several parameters for queuing:

Parameters

Description

Queue capacity limits:

config.queue.max_entries
config.queue.max_bytes
config.queue.max_batch_size
Configure sizes for various aspects of the queue: maximum number of entries, batch size, and queue size in bytes.

When a queue reaches the maximum number of entries and another entry is enqueued, the oldest entry in the queue is deleted to make space for the new entry. The queue code logs a warning when it reaches a capacity threshold of 80% and when it starts to delete entries from the queue, and logs again when the situation normalizes.
Timer usage:

config.queue.concurrency_limit
Only one timer is used to start queue processing in the background by default. Once the queue is empty, the timer handler terminates, and a new timer is created as soon as a new entry is pushed onto the queue.
Retry logic:

config.queue.initial_retry_delay
config.queue.max_coalescing_delay
config.queue.max_retry_delay
config.queue.max_retry_time
If a queue fails to process, it can automatically retry if the failure is temporary, for example due to network problems or upstream unavailability.

A larger config.queue.max_batch_size makes the policy wait for a full batch before it sends. A single request’s metrics are sent to the agent up to config.queue.max_coalescing_delay seconds late. Keep these values low for near-real-time dashboards.

Limitations

At AI Model scope, the Datadog Policy sets the name tag of every metric to ai-gateway, regardless of which AI Model handled the request. The route_name_tag tag doesn’t identify the AI Model either, because it contains a generated route name. As a result, you can’t tell AI Models apart in Datadog by grouping on name.

To tell AI Models apart, create one Datadog Policy per AI Model, attach each policy to its AI Model, and add a tag that identifies the AI Model to each metric in config.metrics, for example model:gpt-4o. Use this workaround only with a policy attached to a single AI Model.

Looking for on-prem configuration? Convert the kongctl configuration to a decK yaml file and use deck file ai2kong. See Configure Kong AI Gateway on-prem for more information.

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!