How to: Verify Upstream TLS

Prerequisites

Kong Konnect

If you don’t have a Konnect account, you can get started quickly with our onboarding wizard.

The following Konnect items are required to complete this tutorial:
- Personal access token (PAT): Create a new personal access token by opening the Konnect PAT page and selecting Generate Token.
Set the personal access token as an environment variable:
```
export KONNECT_TOKEN='YOUR KONNECT TOKEN'
```

Enable the Gateway API

Install the experimental Gateway API CRDs before installing Kong Ingress Controller:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.3.0/experimental-install.yaml

Create a Gateway and GatewayClass instance to use.

echo "
apiVersion: v1
kind: Namespace
metadata:
  name: kong
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: kong
  annotations:
    konghq.com/gatewayclass-unmanaged: 'true'
spec:
  controllerName: konghq.com/kic-gateway-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: kong
spec:
  gatewayClassName: kong
  listeners:
  - name: proxy
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
         from: All
" | kubectl apply -n kong -f -

Create a KIC Control Plane

Use the Konnect API to create a new CLUSTER_TYPE_K8S_INGRESS_CONTROLLER Control Plane:

CONTROL_PLANE_DETAILS=$(curl -X POST "https://us.api.konghq.com/v2/control-planes" \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "name": "My KIC CP",
       "cluster_type": "CLUSTER_TYPE_K8S_INGRESS_CONTROLLER"
     }')

We’ll need the id and telemetry_endpoint for the values.yaml file later. Save them as environment variables:

CONTROL_PLANE_ID=$(echo $CONTROL_PLANE_DETAILS | jq -r .id)
CONTROL_PLANE_TELEMETRY=$(echo $CONTROL_PLANE_DETAILS | jq -r '.config.telemetry_endpoint | sub("https://";"")')

Create mTLS certificates

Kong Ingress Controller talks to Konnect over a connected secured with TLS certificates.

Generate a new certificate using openssl:

openssl req -new -x509 -nodes -newkey rsa:2048 -subj "/CN=kongdp/C=US" -keyout ./tls.key -out ./tls.crt

The certificate needs to be a single line string to send it to the Konnect API with curl. Use awk to format the certificate:

export CERT=$(awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' tls.crt);

Next, upload the certificate to Konnect:

curl -X POST "https://us.api.konghq.com/v2/control-planes/$CONTROL_PLANE_ID/dp-client-certificates" \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "cert": "'$CERT'"
     }'

Finally, store the certificate in a Kubernetes secret so that Kong Ingress Controller can read it:

kubectl create namespace kong -o yaml --dry-run=client | kubectl apply -f -
kubectl create secret tls konnect-client-tls -n kong --cert=./tls.crt --key=./tls.key

Kong Ingress Controller running

Add the Kong Helm charts:

helm repo add kong https://charts.konghq.com
helm repo update

Install Kong Ingress Controller using Helm:

helm install kong kong/ingress -n kong --create-namespace --set controller.ingressController.env.feature_gates="GatewayAlpha=true"

Set $PROXY_IP as an environment variable for future commands:

export PROXY_IP=$(kubectl get svc --namespace kong kong-gateway-proxy -o jsonpath='{range .status.loadBalancer.ingress[0]}{@.ip}{@.hostname}{end}')
echo $PROXY_IP

Required Kubernetes resources

This how-to requires some Kubernetes services to be available in your cluster. These services will be used by the resources created in this how-to.

kubectl apply -f https://developer.konghq.com/manifests/kic/echo-service.yaml -n kong

Generate a CA Certificate

Kong Gateway can validate the certificate chain to a specific depth. To showcase all the possible configurations, create a certificate chain with a root CA, an intermediate CA, and a leaf server certificate:

mkdir certs && cd certs
cd certs

openssl req -new -newkey rsa:2048 -nodes -keyout root.key -subj "/CN=root" -x509 -days 365 -out root.crt

openssl req -new -newkey rsa:2048 -nodes -keyout inter.key -subj "/CN=inter" -out inter.csr
openssl x509 -req -in inter.csr -CA root.crt -CAkey root.key -CAcreateserial -days 365 -out inter.crt -extfile <(echo "basicConstraints=CA:TRUE")

openssl req -new -newkey rsa:2048 -nodes -keyout leaf.key -subj "/CN=kong.example" -out leaf.csr
openssl x509 -req -in leaf.csr -CA inter.crt -CAkey inter.key -CAcreateserial -days 365 -out leaf.crt -extfile <(printf "subjectAltName=DNS:kong.example")

cat leaf.crt inter.crt > chain.crt

rm -f *.csr *.srl
cd ..

Running this script generates the following files in certs directory:

root.key, root.crt: Root CA key and certificate
inter.key, inter.crt: Intermediate CA key and certificate
leaf.key, leaf.crt: Server key and certificate (valid for kong.example SAN)
chain.crt: Server certificate chain

Configure TLS on the echo service

As part of the prerequisites, you deployed the echo Service to your cluster. Let’s configure it to serve HTTPS. Create a secret with the server key and the certificate chain (including the intermediate certificate and the leaf certificate).

Create a Kubernetes secret containing the certificate:

 kubectl create secret -n kong tls goecho-tls --key ./certs/leaf.key --cert ./certs/chain.crt

Patch the echo deployment to use the secret and serve HTTPS using it:

 kubectl patch -n kong deployment echo -p '{
   "spec": {
     "template": {
       "spec": {
         "containers": [
           {
             "name": "echo",
             "ports": [
               {
                 "containerPort": 443
               }
             ],
             "env": [
               {
                 "name": "HTTPS_PORT",
                 "value": "443"
               },
               {
                 "name": "TLS_CERT_FILE",
                 "value": "/etc/tls/tls.crt"
               },
               {
                 "name": "TLS_KEY_FILE",
                 "value": "/etc/tls/tls.key"
               }
             ],
             "volumeMounts": [
               {
                 "mountPath": "/etc/tls",
                 "name": "tls"
               }
             ]
           }
         ],
         "volumes": [
           {
             "name": "tls",
             "secret": {
               "secretName": "goecho-tls"
             }
           }
         ]
       }
     }
   }
 }'

Patch the Service to use HTTPS by adding the konghq.com/protocol: https annotation and the spec.ports entry:

 kubectl patch -n kong service echo -p '{
   "metadata": {
     "annotations": {
       "konghq.com/protocol": "https"
     }
   },
   "spec": {
     "ports": [
       {
         "name": "https",
         "port": 443,
         "targetPort": 443
       }
     ]
   }
 }'

Expose the echo Service

Now that the echo Service is serving an HTTPS endpoint, we need to expose it:

echo "
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: echo
  namespace: kong
  annotations:
    konghq.com/strip-path: 'true'
spec:
  parentRefs:
  - name: kong
    namespace: kong
  hostnames:
  - 'kong.example'
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /echo
    backendRefs:
    - name: echo
      kind: Service
      port: 443
" | kubectl apply -f -

echo "
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: echo
  namespace: kong
  annotations:
    konghq.com/strip-path: 'true'
spec:
  ingressClassName: kong
  rules:
  - host: kong.example
    http:
      paths:
      - path: /echo
        pathType: ImplementationSpecific
        backend:
          service:
            name: echo
            port:
              number: 443
" | kubectl apply -f -

Verify connectivity by making an HTTP request to proxy. The Service serves HTTPS but Kong Gateway initiates the connection and proxies it as HTTP in this case, so the request should be made over HTTP. The Host header has to match the hostname of the Service.

curl "$PROXY_IP/echo" \
     -H "Host: kong.example"

curl "$PROXY_IP/echo" \
     -H "Host: kong.example"

You should see a response similar to this:

Welcome, you are connected to node orbstack.
Running on Pod echo-bd94b7dcc-qxs2b.
In namespace default.
With IP address 192.168.194.9.
Through HTTPS connection.

That means the Service is up and running and Kong Gateway connects to it successfully over HTTPS, without verification.

Configure the root CA Certificate

Before enabling TLS verification, we need to add the root CA certificate to the Kong Gateway’s CA certificates and associate it with the Service.

First, create a Secret with the root CA certificate:

kubectl create -n kong secret generic root-ca \
  --from-file=ca.crt=./certs/root.crt \
  --from-literal=id=bf6e0f14-78cd-45ad-9325-87ec7ef7b892 
kubectl label -n kong secret root-ca konghq.com/ca-cert=true 
kubectl annotate -n kong secret root-ca kubernetes.io/ingress.class=kong

First, create a ConfigMap with the root CA certificate:

kubectl create -n kong configmap root-ca \
  --from-file=ca.crt=./certs/root.crt \
  --from-literal=id=bf6e0f14-78cd-45ad-9325-87ec7ef7b891 
kubectl label -n kong configmap root-ca konghq.com/ca-cert=true 
kubectl annotate -n kong configmap root-ca kubernetes.io/ingress.class=kong

The CA is already associated with the Service through BackendTLSPolicy’s spec.validation.caCertificateRefs.

First, create a Secret with the root CA certificate:

kubectl create -n kong secret generic root-ca \
  --from-file=ca.crt=./certs/root.crt \
  --from-literal=id=bf6e0f14-78cd-45ad-9325-87ec7ef7b892 
kubectl label -n kong secret root-ca konghq.com/ca-cert=true 
kubectl annotate -n kong secret root-ca kubernetes.io/ingress.class=kong

Now, associate the root CA certificate with the Service passing its name to konghq.com/ca-certificates-secrets annotation.

The konghq.com/ca-certificates-secrets annotation is a comma-separated list of Secrets holding CA certificates. You can add multiple Secrets to the list.

kubectl annotate -n kong service echo konghq.com/ca-certificates-secrets='root-ca'

First, create a ConfigMap with the root CA certificate:

kubectl create -n kong configmap root-ca \
  --from-file=ca.crt=./certs/root.crt \
  --from-literal=id=bf6e0f14-78cd-45ad-9325-87ec7ef7b891 
kubectl label -n kong configmap root-ca konghq.com/ca-cert=true 
kubectl annotate -n kong configmap root-ca kubernetes.io/ingress.class=kong

Now, associate the root CA certificate with the Service passing its name to konghq.com/ca-certificates-configmaps annotation.

The konghq.com/ca-certificates-configmaps annotation is a comma-separated list of ConfigMaps holding CA certificates. You can add multiple ConfigMaps to the list.

kubectl annotate -n kong service echo konghq.com/ca-certificates-configmaps='root-ca'

Enable TLS verification

Update your Route to verify the certificate of the upstream service:

Create a BackendTLSPolicy resource:

echo 'apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
  name: goecho-tls-policy
  namespace: kong
spec:
  options:
    tls-verify-depth: "1"
  targetRefs:
  - group: core
    kind: Service
    name: echo
  validation:
    caCertificateRefs:
    - group: core
      kind: Secret # or ConfigMap
      name: root-ca
    hostname: kong.example' | kubectl apply -f -

Annotate the Service with the konghq.com/tls-verify annotation:

kubectl annotate -n kong service echo konghq.com/tls-verify=true

Kong Gateway is now verifying the certificate of the upstream service and accepting the connection because the certificate is trusted.

Configure verification depth

By default, Kong Gateway verifies the certificate chain up to the root CA certificate with no depth limit. You can configure the verification depth by annotating the service with the konghq.com/tls-verify-depth annotation.

To test, set the verification depth to 0 to not allow any intermediate certificates.

kubectl patch -n kong backendtlspolicies.gateway.networking.k8s.io goecho-tls-policy --type merge -p='{
  "spec": {
    "options" : {
      "tls-verify-depth": "0"
    }
  }
}'

kubectl annotate -n kong --overwrite service echo konghq.com/tls-verify-depth=0

Now, when you issue the same request as before, you should see an error stating that an invalid response was received from the upstream server.

By default, Kong Gateway keeps upstream connections alive for 60 seconds (upstream_keepalive_idle_timeout). Due to this, you may need to wait for 60 seconds to see the TLS verification fail. To speed up the process, you can restart the Kong Gateway pod.

curl "$PROXY_IP/echo" \
     -H "Host: kong.example"

curl "$PROXY_IP/echo" \
     -H "Host: kong.example"

{
  "message":"An invalid response was received from the upstream server",
  "request_id":"e2b3182856c96c23d61e880d0a28012f"
}

You can inspect Kong Gateway’s container logs to see the error.

kubectl logs -n kong deploy/kong-gateway | grep "GET /echo"

2024/11/29 11:41:46 [error] 1280#0: *45531 upstream SSL certificate verify error: (22:certificate chain too long) while SSL handshaking to upstream, client: 192.168.194.1, server: kong, request: "GET /echo HTTP/1.1", upstream: "https://192.168.194.19:443/", host: "kong.example", request_id: "678281372fb8907ed06d517cf515de78"
192.168.194.1 - - [29/Nov/2024:11:41:46 +0000] "GET /echo HTTP/1.1" 502 126 "-" "curl/8.7.1" kong_request_id: "678281372fb8907ed06d517cf515de78"

Kong Gateway is now rejecting the connection because the certificate chain is too long. Changing the verification depth to 1 should allow the connection to succeed again.

kubectl patch -n kong backendtlspolicies.gateway.networking.k8s.io goecho-tls-policy --type merge -p='{
  "spec": {
    "options" : {
      "tls-verify-depth": "1"
    }
  }
}'

The results should look like this:

backendtlspolicy.gateway.networking.k8s.io/goecho-tls-policy patched

For example, to limit the verification depth to 1 (i.e., only verify one intermediate certificate), you can annotate the service like this:

kubectl annotate -n kong --overwrite service echo konghq.com/tls-verify-depth=1

Now, when you issue the same request as before, you should see a successful response.

curl "$PROXY_IP/echo" \
     -H "Host: kong.example"

curl "$PROXY_IP/echo" \
     -H "Host: kong.example"

Cleanup

Delete created Kubernetes resources

kubectl delete -n kong -f https://developer.konghq.com/manifests/kic/echo-service.yaml

Uninstall KIC from your cluster

helm uninstall kong -n kong

Verify Upstream TLS