Monitor AI LLM metrics

AI Gateway calls LLM-based services according to the settings of your AI Model Providers and Models. You can use the built in logging and a Prometheus Policy to aggregate the LLM provider responses to count the number of tokens sent through AI Gateway. If you have defined input and output costs in the models, you can also calculate aggregate costs. You can also track whether the requests have been cached by AI Gateway, saving the cost of contacting the LLM providers, which improves performance.

In addition to LLM usage, AI Gateway can also log MCP server traffic. MCP logging provides visibility into latency, response sizes, and error rates when AI Policies invoke external MCP tools and servers.

Create a Prometheus Policy to expose metrics in a Prometheus exposition format, which can be scraped by a Prometheus server.

The Prometheus Policy records and exposes metrics at the node level. Your Prometheus server will need to discover all Kong nodes via a service discovery mechanism, and consume data from each node’s Prometheus /metrics endpoint.

The following sections describe the AI metrics that are available.

LLM traffic metrics

When the config.ai_metrics parameter is set to true in the Prometheus Policy, you can get the following AI LLM metrics:

• AI requests: AI request sent to LLM providers.
• AI cost: AI cost charged by LLM providers.
• AI tokens: AI tokens counted by LLM providers.
• AI LLM latency: Time taken to return a response by LLM providers.
• AI cache fetch latency: Time taken to return a response from the cache.
• AI cache embeddings latency: Time taken to generate embedding during the cache.

These metrics are available per provider, model, cache, database name (if cached), embeddings provider (if cached), embeddings model (if cached), and Workspace. The AI Tokens metrics are also available per token type.

Note: AI metrics include the consumer label. This enables you to attribute AI usage and token counts to individual Consumers, helping you measure cost, performance, and client-specific behavior.

AI metrics (except kong_ai_llm_tokens_total) include the request_mode label. This label shows how the request was processed:

• oneshot: A single response was returned.
• stream: The response was delivered as a stream of tokens.
• realtime: The request was handled as a real-time session.

MCP traffic metrics

When the config.ai_metrics parameter is set to true, the following MCP-specific metrics are also available:

• MCP response body size: Histogram of response body sizes (in bytes) returned by MCP servers.
• MCP latency: Histogram of request latencies (in milliseconds) for MCP server calls.
• MCP error total: Counter of total MCP server errors, labeled by error type.

These metrics are labeled with service, route, method, workspace, and tool_name. The MCP error total metric also includes the type label.

AI metrics are disabled by default, as generating them may create a high number of metrics and affect performance. To enable them, set config.ai_metrics to true in the Prometheus Policy configuration. The AI Model entity automatically collects the statistics that populate these metrics, so no separate AI Model-level configuration is required.

LLM traffic metrics overview

Here is an example of output you could expect from the /metrics endpoint for LLM traffic:

# HELP ai_llm_requests_total AI requests total per ai_provider in Kong
# TYPE ai_llm_requests_total counter
ai_llm_requests_total{ai_provider="provider1",ai_model="model1",cache_status="hit",vector_db="redis",embeddings_provider="openai",embeddings_model="text-embedding-3-large","request_mode"="oneshot",Workspace="workspace1",consumer="consumer1"} 100

# HELP ai_llm_cost_total AI requests cost per ai_provider/cache in Kong
# TYPE ai_llm_cost_total counter
ai_llm_cost_total{ai_provider="provider1",ai_model="model1",cache_status="hit",vector_db="redis",embeddings_provider="openai",embeddings_model="text-embedding-3-large","request_mode"="oneshot",Workspace="workspace1",consumer="consumer1"} 50

# HELP ai_llm_provider_latency AI latencies per ai_provider in Kong
# TYPE ai_llm_provider_latency bucket
ai_llm_provider_latency_ms_bucket{ai_provider="provider1",ai_model="model1",cache_status="",vector_db="",embeddings_provider="",embeddings_model="","request_mode"="oneshot",Workspace="workspace1",le="+Inf",consumer="consumer1"} 2

# HELP ai_llm_tokens_total AI tokens total per ai_provider/cache in Kong
# TYPE ai_llm_tokens_total counter
ai_llm_tokens_total{ai_provider="provider1",ai_model="model1",cache_status="",vector_db="",embeddings_provider="",embeddings_model="",token_type="prompt_tokens",Workspace="workspace1",consumer="consumer1"} 1000
ai_llm_tokens_total{ai_provider="provider1",ai_model="model1",cache_status="",vector_db="",embeddings_provider="",embeddings_model="",token_type="completion_tokens",Workspace="workspace1",consumer="consumer1"} 2000
ai_llm_tokens_total{ai_provider="provider1",ai_model="model1",cache_status="hit",vector_db="redis",embeddings_provider="openai",embeddings_model="text-embedding-3-large",token_type="total_tokens",Workspace="workspace1",consumer="consumer1"} 3000

# HELP ai_cache_fetch_latency AI cache latencies per ai_provider/database in Kong
# TYPE ai_cache_fetch_latency bucket
ai_cache_fetch_latency{ai_provider="provider1",ai_model="model1",cache_status="hit",vector_db="redis",embeddings_provider="openai",embeddings_model="text-embedding-3-large","request_mode"="oneshot",Workspace="workspace1",le="+Inf",consumer="consumer1"} 2

# HELP ai_cache_embeddings_latency AI cache latencies per ai_provider/database in Kong
# TYPE ai_cache_embeddings_latency bucket
ai_cache_embeddings_latency{ai_provider="provider1",ai_model="model1",cache_status="hit",vector_db="redis",embeddings_provider="openai",embeddings_model="text-embedding-3-large","request_mode"="oneshot",Workspace="workspace1",le="+Inf",consumer="consumer1"} 2

# HELP ai_llm_provider_latency AI cache latencies per ai_provider/database in Kong
# TYPE ai_llm_provider_latency bucket
ai_llm_provider_latency{ai_provider="provider1",ai_model="model1",cache_status="hit",vector_db="redis",embeddings_provider="openai",embeddings_model="text-embedding-3-large","request_mode"="oneshot",Workspace="workspace1",le="+Inf",consumer="consumer1"} 2

Note: If you don’t use any caching, then cache_status, vector_db, embeddings_provider, and embeddings_model values will be empty.

To expose the ai_llm_cost_total metric, you must define the model.options.input_cost model.options.output_cost parameters. See the Model configuration reference for more details.

MCP traffic metrics overview

Here is an example of output you could expect from the /metrics endpoint for MCP traffic:

# HELP kong_ai_mcp_response_body_size_bytes MCP server response body sizes in bytes
# TYPE kong_ai_mcp_response_body_size_bytes histogram
kong_ai_mcp_response_body_size_bytes_bucket{service="svc1",route="route1",method="tools/call",workspace="workspace1",tool_name="tool1",le="+Inf"} 1

# HELP kong_ai_mcp_latency_ms MCP server latencies in milliseconds
# TYPE kong_ai_mcp_latency_ms histogram
kong_ai_mcp_latency_ms_bucket{service="svc1",route="route1",method="tools/call",workspace="workspace1",tool_name="tool1",le="+Inf"} 1

# HELP kong_ai_mcp_error_total Total MCP server errors by type
# TYPE kong_ai_mcp_error_total counter
kong_ai_mcp_error_total{service="svc1",route="route1",type="Invalid Request",method="tools/call",workspace="workspace1",tool_name=""} 3

AI Gateway data plane nodes don’t expose an Admin API, so the Status API is the only way to reach the /metrics endpoint. Enable the Status API with the status_listen parameter in the AI Gateway configuration, then point your Prometheus server at each node’s /metrics endpoint.

In most configurations, this endpoint should sit behind a firewall or require authentication, since it isn’t exposed publicly by default.