Set up observability for Kong Event Gateway

Run the quickstart script to automatically provision a demo Event Gateway control plane and data plane, and configure your environment for exporting observability data:

curl -Ls https://get.konghq.com/event-gateway | bash -s -- \
  -k $KONNECT_TOKEN \
  -N kafka_event_gateway \
  -e "OTEL_EXPORTER_OTLP_PROTOCOL=grpc" \
  -e "OTEL_EXPORTER_OTLP_ENDPOINT=http://lgtm:4317" \
  -e "OTEL_EXPORTER_OTLP_TIMEOUT=10s" \
  -e "OTEL_SERVICE_NAME=eventgw"

Where you configure the following custom telemetry settings:

This sets up an Event Gateway control plane named event-gateway-quickstart, provisions a local data plane, and prints out the following environment variable export:

export EVENT_GATEWAY_ID=your-gateway-id

Copy and paste this into your terminal to configure your session.

This quickstart script is meant for demo purposes only, therefore it runs locally with most default parameters and a small number of exposed ports. If you want to run Kong Gateway as a part of a production-ready platform, set up your control plane and data planes through the Konnect UI, or using Terraform.

The grafana/otel-lgtm image bundles Grafana, Tempo, Prometheus, Loki, and a built-in OTEL Collector in a single container. No custom configuration files are needed.

Run the following command to start the LGTM stack on the same network as your Kafka cluster and Event Gateway data plane:

docker run -d --name lgtm \
  --network kafka_event_gateway \
  -p 3000:3000 \
  -p 4317:4317 \
  -p 4318:4318 \
  grafana/otel-lgtm:latest

Use the following Kafka configuration to access your Kafka resources from the virtual clusters:

cat <<EOF > kafkactl.yaml
contexts:
  direct:
    brokers:
      - localhost:9095
      - localhost:9096
      - localhost:9094
  vc:
    brokers:
      - localhost:19092
EOF

Use the following command to create a backend cluster that connects to the Kafka servers you set up:

BACKEND_CLUSTER_ID=$(curl -X POST "https://us.api.konghq.com/v1/event-gateways/$EVENT_GATEWAY_ID/backend-clusters" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "name": "backend_cluster",
       "bootstrap_servers": [
         "kafka1:9092",
         "kafka2:9092",
         "kafka3:9092"
       ],
       "authentication": {
         "type": "anonymous"
       },
       "tls": {
         "enabled": false
       },
       "insecure_allow_anonymous_virtual_cluster_auth": true
     }' | jq -r ".id"
)

In this example configuration:

• bootstrap_servers: Points the backend cluster to the three bootstrap servers that we launched in the prerequisites.
• authentication and insecure_allow_anonymous_virtual_cluster_auth: For demo purposes, we’re allowing insecure anonymous connections, which means no authentication required.
• tls: TLS is disabled so that we can easily test the connection.

Run the following command to create a new virtual cluster associated with our backend cluster. This will let you route event traffic and apply policies:

VIRTUAL_CLUSTER_ID=$(curl -X POST "https://us.api.konghq.com/v1/event-gateways/$EVENT_GATEWAY_ID/virtual-clusters" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "name": "example_virtual_cluster",
       "destination": {
         "id": "'$BACKEND_CLUSTER_ID'"
       },
       "dns_label": "vcluster-1",
       "authentication": [
         {
           "type": "anonymous"
         }
       ],
       "acl_mode": "passthrough"
     }' | jq -r ".id"
)

In this example:

• authentication: Allows anonymous authentication.
• acl_mode: The setting passthrough means that all clients are allowed and don’t have to match a defined ACL. In a production environment, you would set this to enforce_on_gateway and define an ACL policy.
• name is an internal name for the configuration object, while the dns_label is necessary for SNI routing.

For testing purposes, we’ll use port forwarding to route traffic to the virtual cluster.
In production environments, you should use SNI routing instead.

Run the following command to create a new listener:

LISTENER_ID=$(curl -X POST "https://us.api.konghq.com/v1/event-gateways/$EVENT_GATEWAY_ID/listeners" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "name": "example_listener",
       "addresses": [
         "0.0.0.0"
       ],
       "ports": [
         "19092-19095"
       ]
     }' | jq -r ".id"
)

Create the port mapping policy:

curl -X POST "https://us.api.konghq.com/v1/event-gateways/$EVENT_GATEWAY_ID/listeners/$LISTENER_ID/policies" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "type": "forward_to_virtual_cluster",
       "name": "forward",
       "config": {
         "type": "port_mapping",
         "advertised_host": "localhost",
         "destination": {
           "id": "'$VIRTUAL_CLUSTER_ID'"
         }
       }
     }'

Create a topic using the direct context, which is a direct connection to our Kafka cluster:

kafkactl -C kafkactl.yaml --context direct create topic my-test-topic

Then produce a message through the virtual cluster:

kafkactl -C kafkactl.yaml --context vc produce my-test-topic --value="test message"

You should see the following responses:

topic created: my-test-topic
message produced (partition=0	offset=0)

Now that Event Gateway is pushing metrics via OTLP to the LGTM stack, let’s explore them in Grafana.

1. In your browser, open Grafana at http://localhost:3000/.
2. Navigate to Drilldown > Metrics.
3. Search for kong to see the list of available metrics.
4. Let’s look at a sample metric: kong_keg_kafka_backend_roundtrip_duration_seconds_count.

This metric reports how many Kafka backend round-trip observations have been recorded (the number of requests Event Gateway has sent to the backend cluster and received responses for).

On startup, metrics can take up to 5 minutes to start showing up.

Let’s explore the traces generated by Event Gateway.

1. Send any command through the virtual cluster, such as list topics:

   kafkactl -C kafkactl.yaml --context vc list topics
   

   Copied!
2. In your browser, open Grafana at http://localhost:3000/.
3. Navigate to Drilldown > Traces.
4. Select service.name=eventgw and name=request and click Run query.

Here you can see the full trace generated by Event Gateway for each command. For example, you can click a trace to see the span details, including information about the virtual cluster and backend cluster.

Event Gateway also exports logs via OTLP. Let’s explore them in Grafana.

1. In Grafana, navigate to Drilldown > Logs.
2. Use the label filter service_name = eventgw to see Event Gateway logs.