Home / Kong AI Gateway / Kong AI Gateway Policies
Edit
Report

Prometheus Policy

Overview Configuration reference
Related Documentation
AI Gateway Documentation
Made by
Kong Inc.
Incompatible with
on-prem
Scopes
AI Models
AI MCP Servers
AI Agents
AI Consumers
AI Consumer Groups
Global
Minimum Version
AI Gateway - 2.0
Tags
#metrics #prometheus #monitoring
Related Resources
Kong Konnect Observability
OpenTelemetry Policy
Monitor AI LLM metrics

This AI Policy allows you to expose metrics related to AI Gateway and proxied upstream services in Prometheus exposition format, which can be scraped by a Prometheus Server.

Metrics tracked by this AI Policy are available on the Status API at the /metrics endpoint. See Accessing the metrics for more information.

This AI Policy records and exposes metrics at the node level. Your Prometheus server will need to discover all AI Gateway data plane nodes via a Service discovery mechanism, and consume data from each node’s configured /metrics endpoint. This means metrics reflect the internal service and route data model rather than the user-facing entity model.

Grafana dashboard: Metrics exported by this policy can be graphed in Grafana using a drop-in dashboard.

Accessing the metrics

To collect metrics you must enable the Status API on each data plane by passing the KONG_STATUS_LISTEN environment variable, the standard value is KONG_STATUS_LISTEN=0.0.0.0:8100. Your Prometheus instance must be able to reach each data plane on this port over the network.

Configure a Prometheus AI Policy with ai_metrics: true to capture Kong AI Gateway traffic:

curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "name": "my-prometheus-policy",
      "type": "prometheus",
      "display_name": "My Prometheus Policy",
      "config": {
        "ai_metrics": true,
        "status_code_metrics": true,
        "latency_metrics": true
      }
    }
    '

Set your Prometheus configuration to scrape data by targeting your data plane by it’s hostname and the metrics endpoint at the port you set.

For example, a simple Prometheus configuration file could include:

scrape_configs:
- job_name: kong-gateway
  static_configs:
  - targets:
    - ai-quickstart-gateway:8100

Available metrics

You can expose the following metrics:

  • DB reachability: A gauge type with a value of 0 or 1, which represents whether the database can be reached by an AI Gateway node.
  • Connections: Various Nginx connection metrics like active, reading, writing, and number of accepted connections.
  • Data Plane Status: The last seen timestamp, config hash, config sync status, and certificate expiration timestamp for data plane nodes are exported to the control plane.
  • Enterprise License Information: The AI Gateway license expiration date, features and license signature. Those metrics are only exported on self-managed AI Gateway.
  • DB Entity Count: A gauge metric that measures the current number of database entities.
  • Number of Nginx timers: A gauge metric that measures the total number of Nginx timers in a Running or Pending state.
  • AI LLM metrics: AI LLM metrics are available per provider, model, cache, database name (if cached), embeddings provider (if cached), and embeddings model (if cached).

Note: Metrics in Prometheus may be prefixed by a kong label.

Optional metrics

The following metrics are disabled by default as it may create high cardinality of metrics and may cause performance issues.

LLM traffic metrics

When the config.ai_metrics parameter is set to true in the Prometheus Policy, you can get the following AI LLM metrics:

  • AI requests: AI request sent to LLM providers.
  • AI cost: AI cost charged by LLM providers.
  • AI tokens: AI tokens counted by LLM providers.
  • AI LLM latency: Time taken to return a response by LLM providers.
  • AI cache fetch latency: Time taken to return a response from the cache.
  • AI cache embeddings latency: Time taken to generate embedding during the cache.

These metrics are available per provider, model, cache, database name (if cached), embeddings provider (if cached), embeddings model (if cached), and Workspace. The AI Tokens metrics are also available per token type.

Note: AI metrics include the consumer label. This enables you to attribute AI usage and token counts to individual Consumers, helping you measure cost, performance, and client-specific behavior.

AI metrics (except kong_ai_llm_tokens_total) include the request_mode label. This label shows how the request was processed:

  • oneshot: A single response was returned.
  • stream: The response was delivered as a stream of tokens.
  • realtime: The request was handled as a real-time session.

MCP traffic metrics

When the config.ai_metrics parameter is set to true, the following MCP-specific metrics are also available:

  • MCP response body size: Histogram of response body sizes (in bytes) returned by MCP servers.
  • MCP latency: Histogram of request latencies (in milliseconds) for MCP server calls.
  • MCP error total: Counter of total MCP server errors, labeled by error type.

These metrics are labeled with service, route, method, workspace, and tool_name. The MCP error total metric also includes the type label.

Status code metrics

When config.status_code_metrics is set to true:

  • Status codes: HTTP status codes returned by AI Gateway.
    • http_requests_total: HTTP status codes per Consumer/Service/Route at AI Gateway.
    • stream_session_total: Stream status codes per Service/Route in AI Gateway.

Latency metrics

When config.latency_metrics is set to true:

  • Latencies Histograms: Latency (in ms), as measured at AI Gateway:
    • Request: Total time taken by AI Gateway and upstream services to serve requests.
    • AI Gateway: Time taken for AI Gateway to route a request and run all configured policies.
    • Upstream: Time taken by the upstream service to respond to requests.

Bandwidth metrics

When config.bandwidth_metrics is set to true:

  • Bandwidth: Total Bandwidth (egress/ingress) flowing through AI Gateway. This metric is available per Service and as a sum across all Services.

Upstream health metrics

When config.upstream_health_metrics is set to true:

  • Target Health: The healthiness status (healthchecks_off, healthy, unhealthy, or dns_error) of targets that belong to a given upstream as well as their subsystem (http or stream).

Note: Upstream targets’ health information is exported once per subsystem. If both stream and HTTP listeners are enabled, targets’ health will appear twice. Health metrics have a subsystem label to indicate which subsystem the metric refers to.

Metrics output example

Here is an example of output you could expect from the /metrics endpoint:

curl -i http://localhost:8100/metrics

Response:

HTTP/1.1 200 OK
Date: Thu, 17 Sep 2026 16:04:29 GMT
Content-Type: text/plain; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Access-Control-Allow-Origin: *
X-Kong-Status-Request-ID: swA08FHVHQdk3rSJAqf96TKAWjBpYLCf
X-Kong-Admin-Latency: 2
Server: kong/2.1.0-ai-gateway

# HELP kong_ai_llm_provider_latency_ms LLM response Latency for each AI plugins per ai_provider in Kong
# TYPE kong_ai_llm_provider_latency_ms histogram
kong_ai_llm_provider_latency_ms_bucket{ai_provider="openai",ai_model="gpt-4o",cache_status="",vector_db="",embeddings_provider="",embeddings_model="",workspace="default",consumer="",request_mode="oneshot",le="1500"} 1
kong_ai_llm_provider_latency_ms_bucket{ai_provider="openai",ai_model="gpt-4o",cache_status="",vector_db="",embeddings_provider="",embeddings_model="",workspace="default",consumer="",request_mode="oneshot",le="2000"} 1
...
kong_ai_llm_provider_latency_ms_bucket{ai_provider="openai",ai_model="gpt-4o",cache_status="",vector_db="",embeddings_provider="",embeddings_model="",workspace="default",consumer="",request_mode="oneshot",le="+Inf"} 1
kong_ai_llm_provider_latency_ms_count{ai_provider="openai",ai_model="gpt-4o",cache_status="",vector_db="",embeddings_provider="",embeddings_model="",workspace="default",consumer="",request_mode="oneshot"} 1
kong_ai_llm_provider_latency_ms_sum{ai_provider="openai",ai_model="gpt-4o",cache_status="",vector_db="",embeddings_provider="",embeddings_model="",workspace="default",consumer="",request_mode="oneshot"} 1029
# HELP kong_ai_llm_requests_total AI requests total per ai_provider in Kong
# TYPE kong_ai_llm_requests_total counter
kong_ai_llm_requests_total{ai_provider="openai",ai_model="gpt-4o",cache_status="",vector_db="",embeddings_provider="",embeddings_model="",workspace="default",consumer="",request_mode="oneshot"} 1
# HELP kong_ai_llm_tokens_total AI requests cost per ai_provider/cache in Kong
# TYPE kong_ai_llm_tokens_total counter
kong_ai_llm_tokens_total{ai_provider="openai",ai_model="gpt-4o",cache_status="",vector_db="",embeddings_provider="",embeddings_model="",token_type="completion_tokens",workspace="default",consumer=""} 12
kong_ai_llm_tokens_total{ai_provider="openai",ai_model="gpt-4o",cache_status="",vector_db="",embeddings_provider="",embeddings_model="",token_type="prompt_tokens",workspace="default",consumer=""} 13
kong_ai_llm_tokens_total{ai_provider="openai",ai_model="gpt-4o",cache_status="",vector_db="",embeddings_provider="",embeddings_model="",token_type="total_tokens",workspace="default",consumer=""} 25
# HELP kong_ai_llm_tpot_latency_ms LLM time per token latency for each AI plugins per ai_provider in Kong
# TYPE kong_ai_llm_tpot_latency_ms histogram
kong_ai_llm_tpot_latency_ms_bucket{ai_provider="openai",ai_model="gpt-4o",cache_status="",vector_db="",embeddings_provider="",embeddings_model="",workspace="default",consumer="",request_mode="oneshot",le="100"} 1
kong_ai_llm_tpot_latency_ms_bucket{ai_provider="openai",ai_model="gpt-4o",cache_status="",vector_db="",embeddings_provider="",embeddings_model="",workspace="default",consumer="",request_mode="oneshot",le="200"} 1
...
kong_ai_llm_tpot_latency_ms_bucket{ai_provider="openai",ai_model="gpt-4o",cache_status="",vector_db="",embeddings_provider="",embeddings_model="",workspace="default",consumer="",request_mode="oneshot",le="+Inf"} 1
kong_ai_llm_tpot_latency_ms_count{ai_provider="openai",ai_model="gpt-4o",cache_status="",vector_db="",embeddings_provider="",embeddings_model="",workspace="default",consumer="",request_mode="oneshot"} 1
kong_ai_llm_tpot_latency_ms_sum{ai_provider="openai",ai_model="gpt-4o",cache_status="",vector_db="",embeddings_provider="",embeddings_model="",workspace="default",consumer="",request_mode="oneshot"} 85.75
# HELP kong_ai_llm_ttft_latency_ms LLM time to first token latency for each AI plugins per ai_provider in Kong
# TYPE kong_ai_llm_ttft_latency_ms histogram
kong_ai_llm_ttft_latency_ms_bucket{ai_provider="openai",ai_model="gpt-4o",cache_status="",vector_db="",embeddings_provider="",embeddings_model="",workspace="default",consumer="",request_mode="oneshot",le="1500"} 1
kong_ai_llm_ttft_latency_ms_bucket{ai_provider="openai",ai_model="gpt-4o",cache_status="",vector_db="",embeddings_provider="",embeddings_model="",workspace="default",consumer="",request_mode="oneshot",le="2000"} 1
...
kong_ai_llm_ttft_latency_ms_bucket{ai_provider="openai",ai_model="gpt-4o",cache_status="",vector_db="",embeddings_provider="",embeddings_model="",workspace="default",consumer="",request_mode="oneshot",le="+Inf"} 1
kong_ai_llm_ttft_latency_ms_count{ai_provider="openai",ai_model="gpt-4o",cache_status="",vector_db="",embeddings_provider="",embeddings_model="",workspace="default",consumer="",request_mode="oneshot"} 1
kong_ai_llm_ttft_latency_ms_sum{ai_provider="openai",ai_model="gpt-4o",cache_status="",vector_db="",embeddings_provider="",embeddings_model="",workspace="default",consumer="",request_mode="oneshot"} 1029
# HELP kong_control_plane_connected Kong connected to control plane, 0 is unconnected
# TYPE kong_control_plane_connected gauge
kong_control_plane_connected 1
# HELP kong_data_plane_cluster_cert_expiry_timestamp Unix timestamp of Data Plane's cluster_cert expiry time
# TYPE kong_data_plane_cluster_cert_expiry_timestamp gauge
kong_data_plane_cluster_cert_expiry_timestamp 1792252942
# HELP kong_datastore_reachable Datastore reachable from Kong, 0 is unreachable
# TYPE kong_datastore_reachable gauge
kong_datastore_reachable 1
# HELP kong_http_requests_total HTTP status codes per consumer/service/route in Kong
# TYPE kong_http_requests_total counter
kong_http_requests_total{service="ai-gateway",route="openai-chat",code="200",source="service",type="",workspace="default",consumer=""} 1
# HELP kong_kong_internal_latency_ms Internal latency for each service/route in Kong, excluding the I/O latency
# TYPE kong_kong_internal_latency_ms histogram
kong_kong_internal_latency_ms_bucket{service="ai-gateway",route="openai-chat",workspace="default",le="10"} 1
kong_kong_internal_latency_ms_bucket{service="ai-gateway",route="openai-chat",workspace="default",le="15"} 1
kong_kong_internal_latency_ms_bucket{service="ai-gateway",route="openai-chat",workspace="default",le="20"} 1
...

FAQs

If I have multiple data plane nodes, does the Prometheus policy metrics data reflect a single node or an aggregate?

Metrics data is reported per data plane node.

Why do I see kong_nginx_connections_total metrics that aren’t AI related when I’m using the Prometheus policy?

Kong AI Gateway is built on the same foundation as Kong Gateway and some metrics reflect the underlying implementation.

Something wrong?
Report an Issue | Edit this Page
View all Kong documentation

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!
Contribute

Still need help?

Ask in our Slack Community
Contact Support