Autoscale workloads with Prometheus

This tutorial shows how to autoscale workloads based on Service latency. The command service created in the prerequisites allows us to inject an artificial delay in to responses to trigger autoscaling.

The DataPlaneMetricsExtension allows Kong Operator to monitor Service latency and expose it on the /metrics endpoint.

1. Create a DataPlaneMetricsExtension that points to the command service:

    echo '
    kind: DataPlaneMetricsExtension
    apiVersion: gateway-operator.konghq.com/v1alpha1
    metadata:
      name: kong
      namespace: kong
    spec:
      serviceSelector:
        matchNames:
        - name: command
      config:
        latency: true
    ' | kubectl apply -f -

   Copied!

2. Create a GatewayConfiguration that uses it:

    echo '
    kind: GatewayConfiguration
    apiVersion: gateway-operator.konghq.com/v2beta1
    metadata:
      name: kong
      namespace: kong
    spec:
      extensions:
      - kind: DataPlaneMetricsExtension
        group: gateway-operator.konghq.com
        name: kong
    ' | kubectl apply -f -

   Copied!

3. Patch the GatewayClass to use the config:

    kubectl patch -n kong --type=json gatewayclass kong -p='[
        {
            "op":"add",
            "path":"/spec/parametersRef",
            "value":{
                    "group": "gateway-operator.konghq.com",
                    "kind": "GatewayConfiguration",
                    "name": "kong",
                    "namespace": "kong",
            }
        }
    ]'

   Copied!

Note: You can reuse your current Prometheus setup and skip this step but please be aware that it needs to be able to scrape Kong Operator’s metrics (e.g. through ServiceMonitor) and note down the namespace in which it’s deployed.

1. Add the prometheus-community helm charts:

   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
   helm repo update

   Copied!

2. Install Prometheus via kube-prometheus-stack helm chart:

   helm upgrade --install --create-namespace -n prometheus prometheus prometheus-community/kube-prometheus-stack

   Copied!

To make Prometheus scrape Kong Operator’s /metrics endpoint, we’ll need to create a ServiceMonitor:

echo '
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    release: prometheus
  name: gateway-operator
  namespace: kong-system
spec:
  endpoints:
  - port: http
    scheme: http
    path: /metrics
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  selector:
    matchLabels:
      control-plane: controller-manager ' | kubectl apply -f -

After applying the above manifest you can check one of the metrics exposed by Kong Operator to verify that the scrape config has been applied.

To access the Prometheus UI, create a port-forward and visit http://localhost:9090:

kubectl port-forward service/prometheus-kube-prometheus-prometheus 9090:9090 -n prometheus

This can be verified by going to your Prometheus UI and querying:

up{service=~"kong-operator-kong-operator-metrics"}

Prometheus metrics can take up to 2 minutes to appear.

The prometheus-adapter package makes Prometheus metrics usable in Kubernetes.

To deploy prometheus-adapter, you’ll need to decide what time series to expose so that Kubernetes can consume it.

Note: Kong Operator enriches specific metrics for use with prometheus-adapter. See the overview for a complete list.

Create a values.yaml file to deploy the prometheus-adapter helm chart. This configuration calculates a kong_upstream_latency_ms_60s_average metric, which exposes a 60s moving average of upstream response latency:

echo $'
prometheus:
  # Update this value if Prometheus is installed in a different namespace
  url: http://prometheus-kube-prometheus-prometheus.prometheus.svc

rules:
  default: false
  custom:
  - seriesQuery: \'{__name__=~"^kong_upstream_latency_ms_(sum|count)",kubernetes_namespace!="",kubernetes_name!="",kubernetes_kind!=""}\'
    resources:
      overrides:
        exported_namespace:
          resource: "namespace"
        exported_service:
          resource: "service"
    name:
      as: "kong_upstream_latency_ms_60s_average"
    metricsQuery: |
      sum by (exported_service) (rate(kong_upstream_latency_ms_sum{<<.LabelMatchers>>}[60s:10s]))
        /
      sum by (exported_service) (rate(kong_upstream_latency_ms_count{<<.LabelMatchers>>}[60s:10s]))
' > values.yaml

Install prometheus-adapter using Helm:

helm upgrade --install --create-namespace -n prometheus --values values.yaml prometheus-adapter prometheus-community/prometheus-adapter

1. Wait for the Gateway and the command HTTPRoute to be programmed:

   kubectl wait --for=condition=Programmed gateway/kong -n kong --timeout=5m
   kubectl wait --for=jsonpath='{.status.parents[0].conditions[?(@.type=="Programmed")].status}'=True httproute/command -n kong --timeout=5m

   Copied!

2. Get the Gateway’s external IP address:

   export PROXY_IP=$(kubectl get gateway kong -n kong -o jsonpath='{.status.addresses[0].value}')

   Copied!

3. To trigger autoscaling, start sending traffic in the background. Each request makes the underlying deployment sleep for 100ms, which raises the average response time to that value. Leave it running while you work through the rest of this guide: the 60-second average that the autoscaler reads only stays above its target while requests are flowing, and the metric can take a few minutes to be populated. The loop sends a request every second for about 30 minutes, which is long enough for the metric to appear and the autoscaler to react:

   for _ in {1..1800}; do
     curl -k --no-progress-meter --fail-with-body "http://$PROXY_IP/command/shell?cmd=sleep%200.1"
     sleep 1
   done > /dev/null 2>&1 &

   Copied!

When all is configured, you should be able to see the metric you’ve configured in prometheus-adapter exposed via the Kubernetes Custom Metrics API. Wait until the metric is populated, then query it:

timeout 900 bash -c 'until kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/kong/services/command/kong_upstream_latency_ms_60s_average" | grep -q kong_upstream_latency_ms_60s_average; do sleep 10; done'
kubectl get --raw '/apis/custom.metrics.k8s.io/v1beta1/namespaces/kong/services/command/kong_upstream_latency_ms_60s_average' | jq

Note: The prometheus-adapter may take a few minutes to populate the custom metrics. It only discovers kong_upstream_latency_ms_60s_average after the data plane has picked up the metrics extension and started reporting latency, so keep the traffic loop running while you wait.

This should result in:

{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "describedObject": {
        "kind": "Service",
        "namespace": "kong",
        "name": "command",
        "apiVersion": "/v1"
      },
      "metricName": "kong_upstream_latency_ms_60s_average",
      "timestamp": "2024-03-06T13:11:12Z",
      "value": "102312m",
      "selector": null
    }
  ]
}

Note: 102312m is a Kubernetes way of expressing numbers as integers. value represents the latency in microseconds, and is approximately equivalent to 102 milliseconds (ms).

When the metric configured in prometheus-adapter is available through Kubernetes’s Custom Metrics API, we can use it in HorizontalPodAutoscaler to autoscale our workload, specifically the command Deployment.

This can be done by using the following manifest, which will scale the underlying command Deployment between 1 and 10 replicas, trying to keep the average latency across last 60s at 40ms.

echo '
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: command
  namespace: kong
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: command
  minReplicas: 1
  maxReplicas: 10
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 1
      policies:
      - type: Percent
        value: 100
        periodSeconds: 10
    scaleUp:
      stabilizationWindowSeconds: 1
      policies:
      - type: Percent
        value: 100
        periodSeconds: 2
      - type: Pods
        value: 4
        periodSeconds: 2
      selectPolicy: Max

  metrics:
  - type: Object
    object:
      metric:
        name: "kong_upstream_latency_ms_60s_average"
      describedObject:
        apiVersion: v1
        kind: Service
        name: command
      target:
        type: Value
        value: "40" ' | kubectl apply -f -

You can watch SuccessfulRescale events using the following kubectl command:

timeout 60 kubectl get events -n kong --field-selector involvedObject.name=command,involvedObject.kind=HorizontalPodAutoscaler -w | grep -m1 SuccessfulRescale

If everything went well we should see the SuccessfulRescale events:

12m          Normal   SuccessfulRescale   horizontalpodautoscaler/command   New size: 2; reason: Service metric kong_upstream_latency_ms_60s_average above target
12m          Normal   SuccessfulRescale   horizontalpodautoscaler/command   New size: 4; reason: Service metric kong_upstream_latency_ms_60s_average above target
12m          Normal   SuccessfulRescale   horizontalpodautoscaler/command   New size: 8; reason: Service metric kong_upstream_latency_ms_60s_average above target
12m          Normal   SuccessfulRescale   horizontalpodautoscaler/command   New size: 10; reason: Service metric kong_upstream_latency_ms_60s_average above target

Then when latency drops (when the traffic loop finishes, or when you stop it) you should observe the SuccessfulRescale events scaling your workloads down:

4s          Normal   SuccessfulRescale   horizontalpodautoscaler/command   New size: 1; reason: All metrics below target