Active tracing in Konnect

Related Documentation
Incompatible with
on-prem

Active tracing enables Control Plane administrators to initiate targeted “deep tracing” sessions in specific Data Plane nodes. During an active tracing session, the selected Data Plane generates detailed, OpenTelemetry-compatible traces for all requests matching the sampling criteria. The detailed spans are captured for the entire request/response lifecycle. These traces can be visualized with Konnect’s built-in span viewer with no additional instrumentation or telemetry tools.

Kong Konnect’s active tracing capability offers exclusive, in-depth insights that cannot be replicated by third-party telemetry tools. The detailed traces generated during live active tracing session are unique to Kong and provide unparalleled visibility into system performance.

Active tracing adheres to OpenTelemetry naming conventions for spans and attributes, wherever possible, ensuring consistency and interoperability.

Key highlights

  • Traces can be generated for a Service or per Route
  • Refined traces can be generated for all requests matching a sampling criteria
  • Sampling criteria can be defined with simple expressions language, for example: http.method == GET
  • Trace sessions are retained for up to 7 days
  • Traces can be visualized in Konnect’s built in trace viewer

Although active tracing is designed as a debug and troubleshooting tool, it can unlock in-depth insights into the API traffic and serve as a monitoring and observability tool. Under normal conditions, active tracing adds negligible latency. However, under heavy loads, it may affect the throughput.

Reading traces in Konnect trace viewer

Traces captured in an active tracing session can be visualized in Konnect’s built-in trace viewer. The trace viewer displays a Summary view and a Trace view. You can gain instant insights with the summary view while the trace view will help you dive deeper.

Summary view

The summary view helps you visualize the entire API request-response flow in a single glance. This view provides a concise overview of critical metrics and a transaction map. The transaction map includes the plugins executed by Kong Gateway on both the request and the response along with the times spent in each phase. Use the summary view to quickly understand the end-to-end API flow, identify performance bottlenecks, and optimize your API strategy.

Trace view

The trace view gives you unparalleled visibility into Kong Gateway’s internal workings. This detailed view breaks down into individual spans, providing a comprehensive understanding of:

  • Kong Gateway’s internal processes and phases
  • Plugin execution and performance
  • Request and response handling

Use the trace view to troubleshoot issues, optimize performance, and refine your configuration.

Get started with tracing

Active tracing requires the following Data Plane version and environment variables in kong.conf:

  • Version: 3.9.1 or above
  • Environment variables:
    • KONG_CLUSTER_RPC=on
    • KONG_ACTIVE_TRACING=on

Note: Active tracing is currently limited to:

  • Konnect Self-Managed Hybrid Gateways
  • Konnect Dedicated Cloud Gateways
  • Konnect Serverless Gateways

    Active tracing is not supported on Kong Ingress Controller and Kong Native Event Proxy Gateways at this time.

Start a trace session

  1. Navigate to Gateway Manager.
  2. Select a Control Plane which has the Data Plane to be traced.
  3. Click on Active Tracing in left navigation menu.
  4. Click New tracing session, define the criteria and, click Start Session.

Once started, traces will begin to be captured. Click on a trace to visualize it in the trace viewer.

The default session duration is 5 minutes or 200 traces per session. Note the sessions are retained for up to 7 days.

Sampling rules

To capture only the relevant API traffic, use sampling rules. Sampling rules filter and refine the requests to be matched. The matching requests are then traced and captured in the session. There are two options.

  • Basic sampling rules: Allow filtering on Routes and Services.
  • Advanced sampling rules: Specify the desired criteria using expressions. For example, to capture traces for all requests matching 503 response code, specify the following rule:
    http.response.status_code==503
    

Known issues in tech preview

Here is a list of known issues in the tech preview:

  • Incorrect span orders: When spans have very short duration few spans may be displayed in wrong order.
  • Incorrect handling of certain error conditions: Traces may be broken when there are certain error conditions. For example, when DNS name resolution fails.
  • Missing spans during high traffic volumes: When no sampling rule is enabled during a high traffic volume scenario, some traces could be missing spans.

Sample trace

A sample trace is shown below. By inspecting the spans, it’s clear that the bulk of the latency occurs in the pre-function plugin during the access phase.

Active-Tracing Spans

Spans

The following spans are available.

kong

The root span.

This span has the following attributes:

Name

Description

url.full Full url, without query parameters
client.address Remote address of the client making the request. This considers forwarded addresses in cases when a load balancer is in front of Kong. Note: this requires configuring the real_ip_header and trusted_ips global configuration options.
client.port Remote port of the client making the request. This considers forwarded ports in cases when a load balancer is in front of Kong. Note: this requires configuring the real_ip_header and trusted_ips global configuration options.
network.peer.address IP of the component that is connecting to Kong
network.peer.port Port of the component that is connecting to Kong
server.address Kong’s DNS name or IP used in client connection
server.port Kong’s public port
network.protocol.name HTTP, gRPC, WS, Kafka, etc.
http.request.method HTTP request method
http.request.body.size Request content length or equivalent in bytes
http.request.size Request body size and request headers size in bytes
http.response.body.size Response content length or equivalent in bytes
http.response.size Response body size and response headers size in bytes
kong.request.id Unique ID for each request
url.scheme Protocol identifier
network.protocol.version Version of the HTTP protocol used in establishing connection [1.2, 2.0]
tls.client.server_name SNI
http.request.header.host Host header if present. This can be different from the SNI.
proxy.kong.consumer_id Authenticated Consumer ID if present
proxy.kong.upstream_id Resolved Upstream ID
proxy.kong.upstream_status_code status code returned by upstream
http.response.status_code Status code sent back by Kong
proxy.kong.latency.upstream Time between the connection to the upstream and the last byte of response
proxy.kong.latency.total Time between the first byte into Kong and the last byte out of Kong
proxy.kong.latency.internal Time taken by Kong to process the request. Excludes client and upstream read/write times, and i/o times.
proxy.kong.latency.net_io_timings Array containing ip, connect_time, and rw_time. I/o outside of the request context is not considered.
proxy.kong.client_KA Whether the downstream used a KeepAlive connection
tls.resumed Whether the TLS session reused
tls.client.subject x509 client DN (if mTLS)
tls.server.subject x509 DN for cert Kong presented
tls.cipher Negotiated cipher

kong.phase.certificate

A span capturing the execution of the certificate phase of request processing. Any plugins configured for running in this phase will show up as individual child spans.

kong.certificate.plugin.plugin_name

A span capturing the execution of a plugin configured to run in the certificate phase. Multiple such spans can occur in a trace.

This span has the following attributes:

Name

Description

proxy.kong.plugin.instance_id Instance ID of the plugin configuration that ran

kong.read_client_http_headers

A span capturing the time taken to read HTTP headers from the client. This span is useful for detecting clients that are coming over a slow network or a buggy CDN, or simply take too long to send in the HTTP headers.

This span has the following attributes:

Name

Description

proxy.kong.http_headers_count Number of headers sent by the client
proxy.kong.http_headers_size Size (in bytes) of headers sent by the client

kong.read_client_http_body

A span capturing the total time taken to read the full body sent by the client. This span can identify slow clients, a buggy CDN and very large body submissions.

kong.phase.rewrite

A span capturing the execution of the rewrite phase of request processing. Any plugins configured for running in this phase will show up as individual child spans.

kong.rewrite.plugin.plugin_name

A span capturing the execution of a plugin configured to run in the rewrite phase. Multiple such spans can occur in a trace.

This span has the following attributes:

Name

Description

proxy.kong.plugin.instance_id Instance ID of the plugin configuration that ran

kong.io.function

A span capturing network i/o timing that occurs during plugin execution or other request processing.

Can be one of:

  • kong.io.http.request: Requests done by the internal http client during the flow
  • kong.io.http.connect: Connections done by the internal http client during the flow
  • kong.io.redis.function: Redis functions
  • kong.io.socket.function: Functions called on the internal nginx socket

Examples:

  • OIDC plugin making calls to IdP
  • Rate Limiting Advanced plugin making calls to Redis
  • Custom plugins calling HTTP URLs

Multiple instances of this span can occur anywhere in the trace when i/o happens.

This span has the following attributes:

Name

Description

network.peer.address Address of the peer Kong connected with
network.protocol.name Protocol that was used (Redis, TCP, HTTP, gRPC, etc.)

kong.router

A span capturing the execution of the Kong router.

This span has the following attributes:

Name

Description

proxy.kong.router.matched Whether the router find a match for the request
proxy.kong.router.route_id ID of the Route that was matched
proxy.kong.router.service_id ID of the Service that was matched
proxy.kong.router.upstream_path Path of the upstream url returned by the match
proxy.kong.router.cache_hit Whether the match returned from cache

kong.phase.access

A span capturing the execution of the access phase of request processing. Any plugins configured for running in this phase will show up as individual child spans.

kong.access.plugin.plugin_name

A span capturing the execution of a plugin configured to run in the access phase. Multiple such spans can occur in a trace.

This span has the following attributes:

Name

Description

proxy.kong.plugin.instance_id Instance ID of the plugin configuration that ran

kong.dns

A span capturing the time spent in looking up DNS.

This span has the following attributes:

Name

Description

proxy.kong.dns.entry A list of DNS attempts, responses and errors if any

kong.upstream.selection

A span capturing the total time spent in finding a healthy upstream. Depending on configuration, Kong will try to find a healthy upstream by trying various targets in order determined by the load balancing algorithm. Child spans of this span capture the individual attempts.

This span has the following attributes:

Name

Description

proxy.kong.upstream.lb_algorithm the load balancing algorithm used for finding the upstream

kong.upstream.find_upstream

A span capturing the attempt to verify a specific upstream. Kong attempts to open a TCP connection (if not KeepAlive cache is found), do a TLS handshake and send down the HTTP headers. If all of this succeeds, the upstream is healthy and Kong will finish sending the full request and wait for a response. If any of the step fails, Kong will switch to the next target and try again.

This span has the following attributes:

Name

Description

network.peer.address the IP address of the target upstream
network.peer.name the DNS name of the target upstream
network.peer.port the port number of the target
try_count The number of attempts Kong has made to find a healthy upstream
keepalive Is this a KeepAlive connection?

kong.send_request_to_upstream

A span capturing the time taken to finish writing the http request to upstream. This span can be used to identify network delays between Kong and an upstream.

kong.read_headers_from_upstream

A span capturing the time taken for the upstream to generate the response headers. This span can be used to identify slowness in response generation from upstreams.

kong.read_body_from_upstream

A span capturing the time taken for the upstream to generate the response body. This span can be used to identify slowness in response generation from upstreams.

kong.phase.response

A span capturing the execution of the response phase. Any plugins configured for running in this phase will show up as individual child spans. This phase will not run if response streaming is enabled.

kong.response.plugin.plugin_name

A span capturing the execution of a plugin configured to run in the response phase. Multiple such spans can occur in a trace.

This span has the following attributes:

Name

Description

proxy.kong.plugin.instance_id Instance ID of the plugin configuration that ran

kong.phase.header_filter

A span capturing the execution of the header filter phase of response processing. Any plugins configured for running in this phase will show up as individual child spans.

kong.header_filter.plugin.plugin_name

A span capturing the execution of a plugin configured to run in the header_filter phase. Multiple such spans can occur in a trace.

This span has the following attributes:

Name

Description

proxy.kong.plugin.instance_id Instance ID of the plugin configuration that ran

kong.phase.body_filter

A span capturing the execution of the body filter phase of response processing. Any plugins configured for running in this phase will show up as individual child spans.

kong.body_filter.plugin.plugin_name

A span capturing the execution of a plugin configured to run in the body_filter phase. Multiple such spans can occur in a trace.

This span has the following attributes:

Name

Description

proxy.kong.plugin.instance_id Instance ID of the plugin configuration that ran
Something wrong?

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!