Debugging Kong Ingress Controller

If Kong Ingress Controller is behaving in a way that you don’t expect, you can increase the logging to figure what is causing the issue.

To customize the log level of Kong Ingress Controller, set the CONTROLLER_LOG_LEVEL environment variable:

kubectl set env -n kong deployment/kong-controller CONTROLLER_LOG_LEVEL="debug"

Alternatively, you can set this value in your values.yaml file:

controller:
  ingressController:
    env:
      log_level: debug

To view logs, use kubectl logs:

kubectl logs -n kong deployments/kong-controller

Inspecting network traffic allows you to review traffic between the Ingress Controller and the Kong Gateway Admin API and/or between the Kong Gateway proxy and upstream applications. You can use this in situations where logged information doesn’t provide sufficient data on the contents of requests and you want to see exactly what was sent over the network.

Although you cannot install and use tcpdump within the controller or Kong Gateway containers, you can add a tcpdump sidecar to your Pod’s containers. The sidecar will be able to sniff traffic from other containers in the Pod. You can edit your Deployment to add the sidecar to all managed Pods or a single Pod and add the following under the containers section of the Pod spec:

- name: tcpdump
  securityContext:
    runAsUser: 0
  image: corfr/tcpdump
  command:
    - /bin/sleep
    - infinity

kubectl patch --type=json -n kong deployments.apps ingress-kong -p='[{
  "op":"add",
  "path":"/spec/template/spec/containers/-",
  "value":{
    "name":"tcpdump",
    "securityContext":{
        "runAsUser":0
    },
    "image":"corfr/tcpdump",
    "command":["/bin/sleep","infinity"]
  }
}]'

If you are using the Kong Helm chart, you can alternately add this to the sidecarContainers section of values.yaml.

Once the sidecar is running, you can use kubectl exec -it POD_NAME -c tcpdump and run a capture. For example, to capture traffic between the controller and the Admin API:

tcpdump -npi any -s0 -w /tmp/capture.pcap host 127.0.0.1 and port 8001

Or between Kong Gateway and an upstream application with endpoints 10.0.0.50 and 10.0.0.51:

tcpdump -npi any -s0 -w /tmp/capture.pcap host 10.0.0.50 or host 10.0.0.51

Once you’ve replicated the issue, you can stop the capture, exit the container, and use kubectl cp to download the capture from the tcpdump container to a local system for review with Wireshark.

You will typically need to temporarily disable TLS to inspect application-layer traffic. If you have access to the server’s private keys you can decrypt TLS, though this doesn’t work if the session uses an ephemeral cipher (neither the controller nor Kong Gateway proxy support dumping session secrets).

The controller provides access to the Golang profiler, which provides diagnostic information on memory and CPU consumption within the program.

To enable profiling and access it, set CONTROLLER_PROFILING=true in the controller container environment, wait for the Deployment to restart, run kubectl port-forward <POD_NAME> 10256:10256, and visit http://localhost:10256/debug/pprof/.

To enable profiling via Helm, set the following in your values.yaml file:

controller:
  ingressController:
    env:
      profiling: "true"