Home / Kong Operator / Gateway Deployment / How-To
Edit
Report

Autoscale workloads with Datadog

Uses: Kong Operator
Incompatible with
konnect
Related Documentation
Operator Documentation
TL;DR

Deploy a DataPlaneMetricsExtension to collect metrics (like latency) from a target service, expose those metrics on the /metrics endpoint, and configure the operator to reference this data for scaling decisions.

Prerequisites

Kong Operator running

  1. Add the Kong Helm charts:

    helm repo add kong https://charts.konghq.com
helm repo update

  2. Install Kong Operator using Helm:

    helm upgrade --install kong-operator kong/kong-operator -n kong-system \
  --create-namespace \
   --set image.tag=2.3
  kubectl -n kong-system wait --for=condition=Available=true --timeout=300s deployment/kong-operator-kong-operator-controller-manager

If you want cert-manager to issue and rotate the admission and conversion webhook certificates, install cert-manager to your cluster and enable cert-manager integration by passing the following argument while installing, in the next step:

   --set global.webhooks.options.certManager.enabled=true

If you do not enable this, the chart will generate and inject self-signed certificates automatically. We recommend enabling cert-manager to manage the lifecycle of these certificates. Kong Operator needs a certificate authority to sign the certificate for mTLS communication between the control plane and the data plane. This is handled automatically by the Helm chart. If you need to provide a custom CA certificate, refer to the certificateAuthority section in the values.yaml of the Helm chart to learn how to create and reference your own CA certificate.

Apply a KongLicense. This assumes that your license is available in ./license.json

echo "
apiVersion: configuration.konghq.com/v1alpha1
kind: KongLicense
metadata:
 name: kong-license
rawLicenseString: '$(cat ./license.json)'
" | kubectl apply -f -

Enable the Gateway API

The Kong Operator Helm chart installs the Gateway API CRDs when you install it, so you do not need to install them separately.

  1. Create a Gateway and GatewayClass instance to use.

    echo "
apiVersion: v1
kind: Namespace
metadata:
  name: kong
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: kong
  annotations:
    konghq.com/gatewayclass-unmanaged: 'true'
spec:
  controllerName: konghq.com/gateway-operator
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: kong
spec:
  gatewayClassName: kong
  listeners:
  - name: proxy
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: All
" | kubectl apply -n kong -f -

Required Kubernetes resources

This how-to requires some Kubernetes services to be available in your cluster. These services will be used by the resources created in this how-to.

kubectl apply -f https://developer.konghq.com/manifests/kic/command-service.yaml -n kong

This how-to also requires 1 pre-configured route:

echo "
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: command
  namespace: kong
  annotations:
    konghq.com/strip-path: 'true'
spec:
  parentRefs:
  - name: kong
    namespace: kong
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: '/command'
    backendRefs:
    - name: command
      kind: Service
      port: 80
" | kubectl apply -f -

Wait until the pods for the services are ready:

kubectl wait --for=condition=Ready pod --all -n kong --timeout=300s

Get Datadog API and application keys

To install Datadog agents in your cluster you will need:

  • An API key
  • An application key
  • The Datadog site for your region (for example datadoghq.com for the US1 region or datadoghq.eu for the EU region).

Export these to your environment:

export DD_SITE='YOUR DATADOG SITE'
export DD_API_KEY='YOUR DATADOG API KEY'
export DD_APP_KEY='YOUR DATADOG APPLICATION KEY'

This tutorial shows how to autoscale workloads based on Service latency. The command service created in the prerequisites allows us to inject an artificial delay in to responses to trigger autoscaling.

Create a DataPlaneMetricsExtension

The DataPlaneMetricsExtension allows Kong Operator to monitor Service latency and expose it on the /metrics endpoint.

  1. Create a DataPlaneMetricsExtension that points to the command service:

     echo '
 kind: DataPlaneMetricsExtension
 apiVersion: gateway-operator.konghq.com/v1alpha1
 metadata:
   name: kong
   namespace: kong
 spec:
   serviceSelector:
     matchNames:
     - name: command
   config:
     latency: true
 ' | kubectl apply -f -

  2. Create a GatewayConfiguration that uses it:

     echo '
 kind: GatewayConfiguration
 apiVersion: gateway-operator.konghq.com/v2beta1
 metadata:
   name: kong
   namespace: kong
 spec:
   extensions:
   - kind: DataPlaneMetricsExtension
     group: gateway-operator.konghq.com
     name: kong
 ' | kubectl apply -f -

  3. Patch the GatewayClass to use the config:

     kubectl patch -n kong --type=json gatewayclass kong -p='[
     {
         "op":"add",
         "path":"/spec/parametersRef",
         "value":{
                 "group": "gateway-operator.konghq.com",
                 "kind": "GatewayConfiguration",
                 "name": "kong",
                 "namespace": "kong",
         }
     }
 ]'

You can integrate Kong Operator with Datadog Metrics to use Kong Gateway latency metrics to autoscale workloads based on their metrics.

Install Datadog in your Kubernetes cluster

  1. Create the following configuration file:

    echo '
datadog:
  kubelet:
    tlsVerify: false
   
clusterAgent:
  enabled: true
  # Enable the metricsProvider to be able to scale based on metrics in Datadog
  metricsProvider:
    # Set this to true to enable Metrics Provider
    enabled: true
    # Enable usage of DatadogMetric CRD to autoscale on arbitrary Datadog queries
    useDatadogMetrics: true
   
  prometheusScrape:
    enabled: true
    serviceEndpoints: true
   
agents:
  containers:
    agent:
      env:
       - name: DD_HOSTNAME
         valueFrom:
           fieldRef:
             fieldPath: spec.nodeName
 ' > values.yaml

  2. Install Datadog’s helm chart:

     helm repo add datadog https://helm.datadoghq.com
 helm repo update
 helm install -n default datadog -f values.yaml --set datadog.apiKey=${DD_API_KEY} --set datadog.appKey=${DD_APP_KEY} --set datadog.site=${DD_SITE} datadog/datadog

  3. Wait for the DatadogMetric CRD to be established before continuing:

     kubectl wait --for=condition=Established crd/datadogmetrics.datadoghq.com --timeout=120s

Send traffic

To trigger autoscaling, complete the following steps:

  1. Wait for the kong Gateway and the command HTTPRoute to be programmed and for the Gateway to receive an address:

    kubectl wait --for=condition=Programmed gateway/kong -n kong --timeout=5m
kubectl wait --for=jsonpath='{.status.parents[0].conditions[?(@.type=="Programmed")].status}'=True httproute/command -n kong --timeout=5m
kubectl wait --for=jsonpath='{.status.addresses[0].value}' gateway/kong -n kong --timeout=5m

    The Gateway can be Programmed before its address appears in status.addresses. Waiting for the address prevents storing an empty value in PROXY_IP in the next step.

  2. Get the Gateway address and store it in PROXY_IP:

    export PROXY_IP=$(kubectl get gateway kong -n kong -o jsonpath='{.status.addresses[0].value}')

  3. Send 50 requests that cause the underlying deployment to sleep for 100ms on each request, increasing the average response time to that value and building up enough latency data to trigger scaling:

    for _ in {1..50}; do
  curl -k --retry 5 --retry-connrefused --no-progress-meter "http://$PROXY_IP/command/shell?cmd=sleep%200.1"
  sleep 1
done

Annotate Kong Operator with Datadog checks config

Add the following annotation on Kong Operator’s Pod to tell Datadog how to scrape Kong Operator’s metrics:

POD_NAME=$(kubectl get pods -n kong-system -l control-plane=controller-manager -o custom-columns='name:.metadata.name' --no-headers)
kubectl annotate -n kong-system pod $POD_NAME \
  'ad.datadoghq.com/manager.checks={
    "openmetrics": {
      "instances": [
        {
          "prometheus_url": "http://%%host%%:8080/metrics",
          "namespace": "autoscaling",
          "metrics": [
            "kong_upstream_latency_ms"
          ],
          "send_histograms_buckets": true,
          "send_distribution_buckets": true
        }
      ]
    }
  }'

After applying the above you should see avg:autoscaling.kong_upstream_latency_ms{service:command} metrics in your Datadog Metrics explorer.

Expose Datadog metrics to Kubernetes

To use an external metric in HorizontalPodAutoscaler, we need to configure the Datadog agent to expose it.

There are several ways to achieve this but we’ll use a Kubernetes native way and use the DatadogMetric CRD:

echo '
apiVersion: datadoghq.com/v1alpha1
kind: DatadogMetric
metadata:
  name: command-kong-upstream-latency-ms-avg
  namespace: kong
spec:
  query: autoscaling.kong_upstream_latency_ms{service:command} ' | kubectl apply -f -

Note: The Datadog Cluster Agent only starts refreshing a DatadogMetric’s status once it’s referenced by a HorizontalPodAutoscaler. Its ACTIVE/VALID/VALUE fields will stay empty until you create the HPA in the next section.

Use DatadogMetric in HorizontalPodAutoscaler

The command-kong-upstream-latency-ms-avg DatadogMetric from the kong namespace can be used by the Kubernetes HorizontalPodAutoscaler to autoscale our workload, specifically the command Deployment. The HorizontalPodAutoscaler must be created in the same namespace as the command Deployment it targets, which is kong.

  1. Run the following command to scale the underlying command Deployment between 1 and 10 replicas, trying to keep the average latency across last 30s at 40ms:

    echo '
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: command
  namespace: kong
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: command
  minReplicas: 1
  maxReplicas: 10
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 1
      policies:
      - type: Percent
        value: 100
        periodSeconds: 10
    scaleUp:
      stabilizationWindowSeconds: 1
      policies:
      - type: Percent
        value: 100
        periodSeconds: 2
      - type: Pods
        value: 4
        periodSeconds: 2
      selectPolicy: Max
   
  metrics:
  - type: External
    external:
      metric:
        name: datadogmetric@kong:command-kong-upstream-latency-ms-avg
      target:
        type: Value
        value: 40 ' | kubectl apply -f -

  2. Once the HPA is created, wait for DatadogMetric to become active and valid:

     kubectl wait --for=jsonpath='{.status.conditions[?(@.type=="Active")].status}'=True -n kong datadogmetric/command-kong-upstream-latency-ms-avg --timeout=120s
 kubectl wait --for=jsonpath='{.status.conditions[?(@.type=="Valid")].status}'=True -n kong datadogmetric/command-kong-upstream-latency-ms-avg --timeout=120s
 kubectl get -n kong datadogmetric command-kong-upstream-latency-ms-avg

    You should get the following result:

    NAME                                   ACTIVE   VALID   VALUE               REFERENCES          UPDATE TIME
command-kong-upstream-latency-ms-avg   True     True    104.46194839477539  hpa:kong/command   38s

Validate

  1. Run the following command to get the command-kong-upstream-latency-ms-avg metric via the Kubernetes External Metrics API:

     kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/kong/datadogmetric@kong:command-kong-upstream-latency-ms-avg" | jq

    You should get the following result:

    {
  "kind": "ExternalMetricValueList",
  "apiVersion": "external.metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "metricName": "datadogmetric@kong:command-kong-upstream-latency-ms-avg",
      "metricLabels": null,
      "timestamp": "2024-03-08T18:03:02Z",
      "value": "104233138021n"
    }
  ]
}

    Note: 104233138021n is a Kubernetes way of expressing numbers as integers. Since value here represents latency in milliseconds, it is approximately equivalent to 104.23ms.

  2. Check for SuccessfulRescale events:

     kubectl get events -n kong --field-selector involvedObject.name=command,involvedObject.kind=HorizontalPodAutoscaler,reason=SuccessfulRescale --sort-by='.lastTimestamp'

    The result should look like this:

    LAST SEEN   TYPE     REASON              OBJECT                            MESSAGE
38s         Normal   SuccessfulRescale   horizontalpodautoscaler/command   New size: 5; reason: external metric datadogmetric@kong:command-kong-upstream-latency-ms-avg(nil) above target
23s         Normal   SuccessfulRescale   horizontalpodautoscaler/command   New size: 10; reason: external metric datadogmetric@kong:command-kong-upstream-latency-ms-avg(nil) above target

Cleanup

Delete the resources created in this guide

Uninstall the Datadog agent, delete the resources this guide created, then remove the operator. Deleting the kong-system namespace removes the operator:

helm uninstall datadog -n default
kubectl delete -n kong --ignore-not-found \
  horizontalpodautoscaler/command \
  datadogmetric/command-kong-upstream-latency-ms-avg \
  httproute/command \
  gateway/kong \
  gatewayconfiguration/kong
kubectl delete --ignore-not-found gatewayclass/kong
kubectl delete --ignore-not-found namespace kong
helm uninstall kong-operator -n kong-system
kubectl delete --ignore-not-found namespace kong-system
Something wrong?
Report an Issue | Edit this Page
View all Kong documentation

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!
Contribute

Still need help?

Ask in our Slack Community
Contact Support