Debugger spans

Related Documentation
Incompatible with
on-prem
Related Resources

When you set up a tracing session with Konnect Debugger, it collects and reports spans that are part of the request-response flow. The following sections describe the spans available in Konnect Debugger.

On this page, the term “3p” or “3rd party” refers to any network interaction outside the main client → Kong → upstream request flow, such as DNS lookups or calls from Kong to external services like auth providers, Redis, or metadata APIs.

kong

The root span.

This span has the following attributes:

Name

Description

url.full Full url, without query parameters
client.address Remote address of the actual client making the request, including any client that is communicating through a proxy, if possible.

Note: This requires configuring trusted_ips and may require configuring real_ip_header.

client.port Remote port of the client making the request, including any client that is communicating through a proxy, if possible.

Note: This requires configuring trusted_ips and may require configuring real_ip_header.

network.peer.address IP of the network peer that is connecting to Kong
network.peer.port Port of the network peer that is connecting to Kong
server.address Kong Gateway can listen on multiple IP addresses. This is the IP address at which Kong received the request.
server.port The port at which Kong received the request.
network.protocol.name Name of the network protocol. Debugger only supports http.
http.request.method HTTP request method
proxy.kong.request.id Unique ID created by Kong Gateway for each request. Useful for debugging.
proxy.kong.request.time Time taken to process this request as measured from reading the first byte from the client, to writing the last byte to the client. Logging handler time is not part of this calculation.
proxy.kong.http.request.body.size Request body length in bytes. Request header size is not included.
http.request.size Request body size and request headers size in bytes.
proxy.kong.http.response.body.size Response body length in bytes. Response header size is not included.
http.response.size Response body size and response headers size in bytes.
url.scheme Protocol identifier
network.protocol.version Version of the HTTP protocol used in establishing connection [1.2, 2.0].
proxy.kong.request.host The value of Host as determined by Kong Gateway in the following order of precedence:
  1. Hostname from the request line.
  2. Hostname from the Host request header field.
  3. Server name matching a request.
proxy.kong.consumer.id Authenticated Consumer ID, if present.
proxy.kong.upstream.id Resolved Upstream ID.
proxy.kong.upstream.status_code Status code returned by the upstream to Kong Gateway.
http.response.status_code Status code sent back by Kong Gateway to client.
proxy.kong.latency.3p.dns.total_io Time spent performing synchronous DNS I/O operations.
proxy.kong.latency.3p.http_client.total_io Time spent on third party HTTP calls (for example, talking to an auth server or fetching metadata from a external HTTP API).
proxy.kong.latency.3p.redis.total_io Time spent on executing Redis operations and performing network I/O to redis
proxy.kong.latency.3p.tcpsock.total_io Time spent performing raw TCP socket I/O operations that are not using the internal instrumented DNS, HTTP or Redis clients.
proxy.kong.latency.3p.total_io Total time spent performing all third-party I/O operations (DNS + Redis + HTTP + Raw)
proxy.kong.latency.client Time spent waiting for the client to send or receive data. This attribute includes time spent waiting for client to finish TLS handshake (span tls_handshake), send all HTTP headers (span read_client_http_headers), send the request body (span read_client_http_body) and the time taken for the client to read the full response (span wait_for_client_read).
proxy.kong.latency.upstream.read_headers_duration Time spent reading response headers from the upstream.
proxy.kong.latency.upstream.read_body_duration Time spent reading the response body from the upstream.
proxy.kong.latency.upstream Time between the connection to the upstream and the last byte of the response. This factors in attempts to connect to a healthy upstream if multiple upstreams were configured. See the span upstream.selection to see a breakdown of each failed attempt and the successful attempt.
proxy.kong.latency.total Time between the first byte into Kong and the last byte out of Kong
proxy.kong.latency.internal Time taken by Kong Gateway to process the request. Excludes client and upstream read/write times, and third party I/O. This is the actual contribution of Kong Gateway itself to the latency of the request.
http.request.header.connection The value of the Connection header if present. A value of close implies that the client does not want to use KeepAlive.
proxy.kong.client.connection.request_count The total number of HTTP requests sent by the client over the same reused underlying TCP connection.
tls.resumed Whether the TLS session was resumed. A value of true implies that the client is reusing an existing TLS (and TCP) connection.
tls.client.subject x509 client DN (if mTLS).
tls.server.subject x509 DN for cert Kong presented.
tls.cipher Negotiated cipher.

kong.phase.certificate

A span capturing the execution of the certificate phase of request processing. Any plugins configured for running in this phase will show up as individual child spans. This phase runs before the TLS handshake. Plugins usually use this phase to set up the server certificate they want to present to clients.

kong.certificate.plugin.plugin_name

A span capturing the execution of a plugin configured to run in the certificate phase. Multiple such spans can occur in a trace.

This span has the following attributes:

Name

Description

proxy.kong.plugin.id ID of the plugin configuration that ran.
proxy.kong.plugin.http.response.status_code v3.14+ HTTP status code set by the plugin when exiting early via kong.response.exit(). Only present when the plugin terminates the request.
proxy.kong.plugin.http.response.status_message v3.14+ HTTP status message set by the plugin when exiting early via kong.response.exit(). Only present when the plugin terminates the request.

kong.tls_handshake

A span that captures the TLS handshake between the client and Kong. This span includes any I/O operations involved in the handshake. Long span times are almost always indicative of slow clients, clients that are many hops away or are connecting over high latency connections.

kong.read_client_http_headers

A span capturing the time taken to read HTTP headers from the client. This span is useful for detecting clients that are coming over a slow network or a buggy CDN, or simply take too long to send in the HTTP headers.

This span has the following attributes:

Name

Description

proxy.kong.http_headers_count Number of headers sent by the client
proxy.kong.http_headers_size Size (in bytes) of headers sent by the client

kong.read_client_http_body

A span capturing the total time taken to read the full body sent by the client. This span can identify slow clients, a buggy CDN and very large body submissions.

kong.phase.rewrite

A span capturing the execution of the rewrite phase of request processing. Any plugins configured for running in this phase will show up as individual child spans.

kong.rewrite.plugin.plugin_name

A span capturing the execution of a plugin configured to run in the rewrite phase. Multiple such spans can occur in a trace.

This span has the following attributes:

Name

Description

proxy.kong.plugin.id ID of the plugin configuration that ran.
proxy.kong.plugin.http.response.status_code v3.14+ HTTP status code set by the plugin when exiting early via kong.response.exit(). Only present when the plugin terminates the request.
proxy.kong.plugin.http.response.status_message v3.14+ HTTP status message set by the plugin when exiting early via kong.response.exit(). Only present when the plugin terminates the request.

kong.io..

A span capturing network i/o timing that occurs during plugin execution or other request processing.

Can be one of:

  • kong.io.http.request: Requests done by the internal http client during the flow
  • kong.io.http.connect: Connections done by the internal http client during the flow
  • kong.io.redis.<function>: Redis functions
  • kong.io.socket.connect: Connections on the internal Nginx socket
  • kong.io.socket.sslhandshake: SSL handshake operations on the internal Nginx socket
  • kong.io.socket.send: Send operations on the internal Nginx socket
  • kong.io.socket.receive: Receive operations on the internal Nginx socket

Examples:

  • OIDC plugin making calls to IdP
  • Rate Limiting Advanced plugin making calls to Redis
  • Custom plugins calling HTTP URLs

Multiple instances of this span can occur anywhere in the trace when i/o happens.

This span has the following attributes:

Name

Description

network.peer.address Address of the peer Kong connected with
network.protocol.name Protocol that was used (Redis, TCP, HTTP, gRPC, etc.)

kong.router

A span capturing the execution of the Kong Gateway router.

This span has the following attributes:

Name

Description

proxy.kong.router.matched Whether the router find a match for the request
proxy.kong.router.route_id ID of the Route that was matched
proxy.kong.router.service_id ID of the Service that was matched
proxy.kong.router.upstream_path Path of the upstream url returned by the match
proxy.kong.router.cache_hit Whether the match returned from cache

kong.phase.access

A span capturing the execution of the access phase of request processing. Any plugins configured for running in this phase will show up as individual child spans.

kong.access.plugin.plugin_name

A span capturing the execution of a plugin configured to run in the access phase. Multiple such spans can occur in a trace.

This span has the following attributes:

Name

Description

proxy.kong.plugin.id ID of the plugin configuration that ran.
proxy.kong.plugin.http.response.status_code v3.14+ HTTP status code set by the plugin when exiting early via kong.response.exit(). Only present when the plugin terminates the request.
proxy.kong.plugin.http.response.status_message v3.14+ HTTP status message set by the plugin when exiting early via kong.response.exit(). Only present when the plugin terminates the request.

kong.dns

A span capturing the time spent in looking up DNS.

This span has the following attributes:

Name

Description

proxy.kong.dns.domain The domain name that was looked up
proxy.kong.dns.tries A list of DNS attempts, responses and errors if any

kong.upstream.selection

A span capturing the total time spent in finding a healthy upstream. Depending on configuration, Kong will try to find a healthy upstream by trying various targets in order determined by the load balancing algorithm. Child spans of this span capture the individual attempts.

This span has the following attributes:

Name

Description

proxy.kong.upstream.lb_algorithm the load balancing algorithm used for finding the upstream

kong.find_upstream

A span capturing the attempt to verify a specific upstream. Kong Gateway attempts to open a TCP connection (if not KeepAlive cache is found), and performs a TLS handshake if the upstream is configured to use TLS. If all of this succeeds, the upstream is healthy and Kong will finish sending the full request and wait for a response. If any of the step fails, Kong Gateway will switch to the next target and try again.

The last of these spans (or the only one, if the first attempt succeeds) ends as soon as the connection is established, ensuring that the total time captured by the parent kong.upstream.selection span always reflects only the time spent connecting to the selected upstream.

Depending on how the proxy_next_upstream directive is configured, earlier (failed) tries may involve additional I/O. For example, if retries are triggered based on the upstream’s status code or header validity, those attempts will include sending the request and reading the response status line and headers which is enough for Kong to determine whether to retry.

This span has the following attributes:

Name

Description

network.peer.address the IP address of the target upstream
network.peer.name the DNS name of the target upstream
network.peer.port the port number of the target
try_count The number of attempts Kong has made to find a healthy upstream
keepalive Checks if it is a KeepAlive connection.

kong.send_request_to_upstream

A span capturing the time taken to write the http request to the upstream. This span can be used to identify network delays between Kong and an upstream.

kong.read_headers_from_upstream

A span capturing the time taken for the upstream to generate the response headers. This span can be used to identify slowness in response generation from upstreams. If there is a delay after the request is sent but before the upstream starts responding, that time to first byte is also included in this span.

kong.read_body_from_upstream

A span capturing the time taken for the upstream to generate the response body. This span can be used to identify slowness in response generation from upstreams.

kong.phase.response

A span capturing the execution of the response phase. Any plugins configured for running in this phase will show up as individual child spans. This phase will not run if response streaming is enabled.

kong.response.plugin.plugin_name

A span capturing the execution of a plugin configured to run in the response phase. Multiple such spans can occur in a trace.

This span has the following attributes:

Name

Description

proxy.kong.plugin.id ID of the plugin configuration that ran.
proxy.kong.plugin.http.response.status_code v3.14+ HTTP status code set by the plugin when exiting early via kong.response.exit(). Only present when the plugin terminates the request.
proxy.kong.plugin.http.response.status_message v3.14+ HTTP status message set by the plugin when exiting early via kong.response.exit(). Only present when the plugin terminates the request.

kong.phase.header_filter

A span capturing the execution of the header filter phase of response processing. Any plugins configured for running in this phase will show up as individual child spans.

kong.header_filter.plugin.plugin_name

A span capturing the execution of a plugin configured to run in the header_filter phase. Multiple such spans can occur in a trace.

This span has the following attributes:

Name

Description

proxy.kong.plugin.id ID of the plugin configuration that ran.
proxy.kong.plugin.http.response.status_code v3.14+ HTTP status code set by the plugin when exiting early via kong.response.exit(). Only present when the plugin terminates the request.
proxy.kong.plugin.http.response.status_message v3.14+ HTTP status message set by the plugin when exiting early via kong.response.exit(). Only present when the plugin terminates the request.

kong.phase.body_filter

A span capturing the execution of the body filter phase of response processing. Any plugins configured for running in this phase will show up as individual child spans.

kong.body_filter.plugin.plugin_name

A span capturing the execution of a plugin configured to run in the body_filter phase. Multiple such spans can occur in a trace.

This span has the following attributes:

Name

Description

proxy.kong.plugin.id ID of the plugin configuration that ran.
proxy.kong.plugin.http.response.status_code v3.14+ HTTP status code set by the plugin when exiting early via kong.response.exit(). Only present when the plugin terminates the request.
proxy.kong.plugin.http.response.status_message v3.14+ HTTP status message set by the plugin when exiting early via kong.response.exit(). Only present when the plugin terminates the request.

kong.wait_for_client_read

A span that measures the time Kong spends finishing the response write to the client. This duration may be extended for slow-reading clients, resulting in a longer span.

kong.phase.log

A span capturing the execution of the log phase. Any plugins configured for running in this phase will show up as individual child spans.

kong.analytics.log_request

A span capturing the execution of analytics logging. This span represents the time taken to prepare and serialize analytics data during each request’s log phase. That data will be later exported to Konnect in batches, asynchronously.

kong.log.plugin.plugin_name

A span capturing the execution of a plugin configured to run in the log phase. Multiple such spans can occur in a trace.

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!