Set up Dynatrace with OpenTelemetry

Premium Partner

Uses: Kong Gateway decK

Deployment Platform:

Prerequisites

Tracing environment variables

Set the following Dynatrace tracing variables before you configure the Data Plane:

export KONG_TRACING_INSTRUMENTATIONS=all
export KONG_TRACING_SAMPLING_RATE=1.0

When you create the Data Plane in Konnect, add the following Kong Gateway configuration variables for Dynatrace tracing:

-e "KONG_TRACING_INSTRUMENTATIONS=all" \
-e "KONG_TRACING_SAMPLING_RATE=1.0" \

Kong Konnect

This is a Konnect tutorial. If you don’t have a Konnect account, you can get started quickly with our onboarding wizard.

The following Konnect items are required to complete this tutorial:
- Personal access token (PAT): Create a new personal access token by opening the Konnect PAT page and selecting Generate Token.
- Control Plane Name: You can use an existing Control Plane or create a new one to use for this tutorial.
- Konnect Proxy URL: By default, a self-hosted Data Plane uses http://localhost:8000. You can set up Data Plane nodes for your Control Plane from the Gateway Manager in Konnect.

Set the personal access token, the Control Plane name, the Control Plane URL, and the Konnect proxy URL as environment variables:

 export DECK_KONNECT_TOKEN='YOUR KONNECT TOKEN'
 export DECK_KONNECT_CONTROL_PLANE_NAME='YOUR CONTROL PLANE NAME'
 export KONNECT_CONTROL_PLANE_URL=https://us.api.konghq.com
 export KONNECT_PROXY_URL='KONNECT PROXY URL'

Kong Gateway running

This tutorial requires Kong Gateway Enterprise. If you don’t have Kong Gateway set up yet, you can use the quickstart script with an enterprise license to get an instance of Kong Gateway running almost instantly.

Export your license to an environment variable:

 export KONG_LICENSE_DATA='LICENSE-CONTENTS-GO-HERE'

Run the quickstart script:

 curl -Ls https://get.konghq.com/quickstart | bash -s -- -e KONG_LICENSE_DATA \
      -e KONG_TRACING_INSTRUMENTATIONS \
      -e KONG_TRACING_SAMPLING_RATE

Once Kong Gateway is ready, you will see the following message:

 Kong Gateway Ready

decK

decK is a CLI tool for managing Kong Gateway declaratively with state files. To complete this tutorial you will first need to install decK.

Required entities

For this tutorial, you’ll need Kong Gateway entities, like Gateway Services and Routes, pre-configured. These entities are essential for Kong Gateway to function but installing them isn’t the focus of this guide. Follow these steps to pre-configure them:

Run the following command:

echo '
_format_version: "3.0"
services:
  - name: example-service
    url: http://httpbin.konghq.com/anything
routes:
  - name: example-route
    paths:
    - "/anything"
    service:
      name: example-service
' | deck gateway apply -

To learn more about entities, you can read our entities documentation.

Dynatrace

This tutorial requires you to have a Dynatrace SaaS account.

In Dynatrace, find your environment ID.
Generate an API token with the openTelemetryTrace.ingest and metrics.ingest scopes.

Export those values as environment variables:

export DECK_DYNATRACE_ENVIRONMENT_ID=<environment-id-here>
export DECK_DYNATRACE_API_TOKEN=<token-here>

Enable the OTEL plugin

In this tutorial, we’ll be configuring the OpenTelemetry plugin to send Kong Gateway traces and logs to Dynatrace SaaS. This configuration is good for testing purposes, but we recommend using a collector, like Dynatrace Collector, in production environments.

Enable the OTEL plugin with Dynatrace settings configured:

echo '
_format_version: "3.0"
plugins:
  - name: opentelemetry
    config:
      traces_endpoint: https://${{ env "DECK_DYNATRACE_ENVIRONMENT_ID" }}.live.dynatrace.com/api/v2/otlp/v1/traces
      logs_endpoint: https://${{ env "DECK_DYNATRACE_ENVIRONMENT_ID" }}.live.dynatrace.com/api/v2/otlp/v1/logs
      resource_attributes:
        service.name: kong-dev
      headers:
        Authorization: Api-Token ${{ env "DECK_DYNATRACE_API_TOKEN" }}
' | deck gateway apply -

Validate

Send a POST request to generate traffic that we can use to validate that Dynatrace is receiving the traces:

curl -X POST "$KONNECT_PROXY_URL/anything" \
     -H "Accept: application/json"\
     -H "Content-Type: application/json"

curl -X POST "http://localhost:8000/anything" \
     -H "Accept: application/json"\
     -H "Content-Type: application/json"

In the Dynatrace UI, navigate to Distributed Traces and search for Service name of kong-dev. You should see a trace for the request you just sent. Sometimes it can take a few seconds to display.

Cleanup

Clean up Konnect environment

If you created a new control plane and want to conserve your free trial credits or avoid unnecessary charges, delete the new control plane used in this tutorial.

Destroy the Kong Gateway container

curl -Ls https://get.konghq.com/quickstart | bash -s -- -d

FAQs

How do I export application span metrics?

Kong Gateway relies on the OpenTelemetry Collector to calculate the metrics based on the traces the OpenTelemetry plugin generates.

To include span metrics for application traces, configure the collector exporters section of the OpenTelemetry Collector configuration file:

connectors:
spanmetrics:
  dimensions:
    - name: http.method
      default: GET
    - name: http.status_code
    - name: http.route
  exclude_dimensions:
    - status.code
  metrics_flush_interval: 15s
  histogram:
    disable: false

service:
pipelines:
  traces:
    receivers: [otlp]
    processors: []
    exporters: [spanmetrics]
  metrics:
    receivers: [spanmetrics]
    processors: []
    exporters: [otlphttp]

I want to use the Dynatrace Collector between Kong Gateway and Dynatrace, how do I configure that?

Make an otel-collector-config.yaml file with the following configuration:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

exporters:
  otlphttp:
    endpoint: "https://{yourEnvironmentId}.live.dynatrace.com/api/v2/otlp"
    headers: 
      "Authorization": "Api-Token <your-api-token>"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: []
      exporters: [otlphttp]
    logs:
      receivers: [otlp]
      processors: []
      exporters: [otlphttp]

I’m getting a 200 OK when I make requests, but nothing is showing up in Dynatrace, how do I fix this?

Make sure your API token in Dynatrace has the openTelemetryTrace.ingest and metrics.ingest scopes. Sometimes it can take a few minutes for the traces to display in Dynatrace.

I’m getting a Missing authorization parameter., context: ngx.timer in Kong Gateway logs when I send a request after configuring the OpenTelemetry plugin with Dynatrace, how do I fix this?

This error is because you need to add the Dynatrace API token as an Authorization header when you configure the OpenTelemetry plugin.