Set up Jaeger with OpenTelemetry

Uses: Kong Gateway decK
Minimum Version
Kong Gateway - 3.4
TL;DR

You can use the OpenTelemetry plugin with Jaeger to send analytics and monitoring data to Jaeger dashboards. Set KONG_TRACING_INSTRUMENTATIONS=all and KONG_TRACING_SAMPLING_RATE=1.0. Enable the OTEL plugin with your Jaeger tracing endpoint, and specify the name you want to track the traces by in resource_attributes.service.name.

Prerequisites

Set the following Jaeger tracing variables before you configure the Data Plane:

export KONG_TRACING_INSTRUMENTATIONS=all
export KONG_TRACING_SAMPLING_RATE=1.0

When you create the Data Plane in Konnect, add the following Kong Gateway configuration variables for Jaeger tracing:

-e "KONG_TRACING_INSTRUMENTATIONS=all" \
-e "KONG_TRACING_SAMPLING_RATE=1.0" \

This is a Konnect tutorial. If you don’t have a Konnect account, you can get started quickly with our onboarding wizard.

  1. The following Konnect items are required to complete this tutorial:

    • Personal access token (PAT): Create a new personal access token by opening the Konnect PAT page and selecting Generate Token.
    • Control Plane Name: You can use an existing Control Plane or create a new one to use for this tutorial.
    • Konnect Proxy URL: By default, a self-hosted Data Plane uses http://localhost:8000. You can set up Data Plane nodes for your Control Plane from the Gateway Manager in Konnect.
  2. Set the personal access token, the Control Plane name, the Control Plane URL, and the Konnect proxy URL as environment variables:

     export DECK_KONNECT_TOKEN='YOUR KONNECT TOKEN'
     export DECK_KONNECT_CONTROL_PLANE_NAME='YOUR CONTROL PLANE NAME'
     export KONNECT_CONTROL_PLANE_URL=https://us.api.konghq.com
     export KONNECT_PROXY_URL='KONNECT PROXY URL'
    

This tutorial requires Kong Gateway Enterprise. If you don’t have Kong Gateway set up yet, you can use the quickstart script with an enterprise license to get an instance of Kong Gateway running almost instantly.

  1. Export your license to an environment variable:

     export KONG_LICENSE_DATA='LICENSE-CONTENTS-GO-HERE'
    
  2. Run the quickstart script:

     curl -Ls https://get.konghq.com/quickstart | bash -s -- -e KONG_LICENSE_DATA \
          -e KONG_TRACING_INSTRUMENTATIONS \
          -e KONG_TRACING_SAMPLING_RATE
    

    Once Kong Gateway is ready, you will see the following message:

     Kong Gateway Ready
    

decK is a CLI tool for managing Kong Gateway declaratively with state files. To complete this tutorial you will first need to install decK.

For this tutorial, you’ll need Kong Gateway entities, like Gateway Services and Routes, pre-configured. These entities are essential for Kong Gateway to function but installing them isn’t the focus of this guide. Follow these steps to pre-configure them:

  1. Run the following command:

    echo '
    _format_version: "3.0"
    services:
      - name: example-service
        url: http://httpbin.konghq.com/anything
    routes:
      - name: example-route
        paths:
        - "/anything"
        service:
          name: example-service
    ' | deck gateway apply -
    

To learn more about entities, you can read our entities documentation.

This tutorial requires you to install Jaeger.

In a new terminal window, deploy a Jaeger instance with Docker in all-in-one mode:

docker run --rm --name jaeger \
-e COLLECTOR_OTLP_ENABLED=true \
-p 16686:16686 \
-p 4317:4317 \
-p 4318:4318 \
-p 5778:5778 \
-p 9411:9411 \
jaegertracing/jaeger:2.5.0

The COLLECTOR_OTLP_ENABLED environment variable must be set to true to enable the OpenTelemetry Collector.

In this tutorial, we’re using host.docker.internal as our host instead of the localhost that Jaeger is using because Kong Gateway is running in a container that has a different localhost to you. Export the host as an environment variable in the terminal window you used to set the other Kong Gateway environment variables:

export DECK_JAEGER_HOST=host.docker.internal

Enable the OTEL plugin

In this tutorial, we’ll be configuring the OpenTelemetry plugin to send Kong Gateway traces to Jaeger.

Enable the OTEL plugin with Jaeger settings configured:

echo '
_format_version: "3.0"
plugins:
  - name: opentelemetry
    config:
      traces_endpoint: http://${{ env "DECK_JAEGER_HOST" }}:4318/v1/traces
      resource_attributes:
        service.name: kong-dev
' | deck gateway apply -

For more information about the ports Jaeger uses, see API Ports in the Jaeger documentation.

Validate

Send a POST request to generate traffic that we can use to validate that Jaeger is receiving the traces:

curl -X POST "$KONNECT_PROXY_URL/anything" \
     -H "Accept: application/json"\
     -H "Content-Type: application/json"
curl -X POST "http://localhost:8000/anything" \
     -H "Accept: application/json"\
     -H "Content-Type: application/json"

In the Jaeger UI, search for kong-dev in Service and click Find Traces. You should see a trace for the request you just sent. Sometimes it can take a few seconds to display.

Cleanup

If you created a new control plane and want to conserve your free trial credits or avoid unnecessary charges, delete the new control plane used in this tutorial.

curl -Ls https://get.konghq.com/quickstart | bash -s -- -d

FAQs

Create a config file (otelcol.yaml) for the OpenTelemetry Collector:

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
  logging:
    loglevel: debug
  zipkin:
    endpoint: "http://some.url:9411/api/v2/spans"
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, zipkin]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging]

Run the OpenTelemetry Collector with Docker:

docker run --name opentelemetry-collector \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 55679:55679 \
  -v $(pwd)/otelcol.yaml:/etc/otel-collector-config.yaml \
  otel/opentelemetry-collector-contrib:0.52.0 \
  --config=/etc/otel-collector-config.yaml

See the OpenTelemetry Collector documentation for more information. Now you can enable the OTEL plugin.

Something wrong?

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!