Home / How-to Guides
Edit
Report

Progressively roll in strict mTLS with Kong Mesh

Uses: Kong Mesh
Related Documentation
Mesh Documentation
Related Resources
MeshTLS policy
Resource sizing guidelines
Version compatibility
Minimum Version
Kong Mesh - 2.9
TL;DR

Create an app in a namespace outside of the mesh, create a permissive MeshTLS policy to allow the Service to receive traffic during the migration, then migrate the app to the mesh and update the MeshTLS to use strict mode.

Prerequisites

Helm

You will need Helm, a package manager for Kubernetes.

A running Kubernetes cluster

This guide requires a running Kubernetes cluster. If you already have a Kubernetes cluster running, you can skip this step. It can be a cluster running locally, like Docker, or in a public cloud like AWS EKS, GCP GKE, etc.

For example, if you are using minikube:

minikube start -p mesh-zone

Install Kong Mesh with demo configuration

  1. Install Kong Mesh:

    helm repo add kong-mesh https://kong.github.io/kong-mesh-charts
helm repo update
helm upgrade \
  --install \
  --create-namespace \
  --namespace kong-mesh-system \
  kong-mesh kong-mesh/kong-mesh
kubectl wait -n kong-mesh-system --for=condition=ready pod --selector=app=kong-mesh-control-plane --timeout=90s

  2. Apply the demo configuration:

    echo "
apiVersion: v1
kind: Namespace
metadata:
  labels:
    kuma.io/sidecar-injection: enabled
  name: kong-mesh-demo
---
apiVersion: v1
kind: Service
metadata:
  name: demo-app
  namespace: kong-mesh-demo
spec:
  ports:
  - appProtocol: http
    port: 5050
    protocol: TCP
    targetPort: 5050
  selector:
    app: demo-app
---
apiVersion: v1
kind: Service
metadata:
  name: demo-app-v1
  namespace: kong-mesh-demo
spec:
  ports:
  - appProtocol: http
    port: 5050
    protocol: TCP
    targetPort: 5050
  selector:
    app: demo-app
    version: v1
---
apiVersion: v1
kind: Service
metadata:
  name: demo-app-v2
  namespace: kong-mesh-demo
spec:
  ports:
  - appProtocol: http
    port: 5050
    protocol: TCP
    targetPort: 5050
  selector:
    app: demo-app
    version: v2
---
apiVersion: v1
kind: Service
metadata:
  name: kv
  namespace: kong-mesh-demo
spec:
  ports:
  - appProtocol: http
    port: 5050
    protocol: TCP
    targetPort: 5050
  selector:
    app: kv
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: demo-app
    version: v1
  name: demo-app
  namespace: kong-mesh-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demo-app
      version: v1
  template:
    metadata:
      labels:
        app: demo-app
        version: v1
    spec:
      containers:
      - env:
        - name: OTEL_SERVICE_NAME
          value: demo-app
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: http://opentelemetry-collector.mesh-observability:4317
        - name: KV_URL
          value: http://kv.kong-mesh-demo.svc.cluster.local:5050
        - name: APP_VERSION
          valueFrom:
            fieldRef:
              fieldPath: metadata.labels['version']
        image: ghcr.io/kumahq/kuma-counter-demo:latest@sha256:daf8f5cffa10b576ff845be84e4e3bd5a8a6470c7e66293c5e03a148f08ac148
        name: app
        ports:
        - containerPort: 5050
          name: http
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: demo-app
    version: v2
  name: demo-app-v2
  namespace: kong-mesh-demo
spec:
  replicas: 0
  selector:
    matchLabels:
      app: demo-app
      version: v2
  template:
    metadata:
      labels:
        app: demo-app
        version: v2
    spec:
      containers:
      - env:
        - name: OTEL_SERVICE_NAME
          value: demo-app
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: http://opentelemetry-collector.mesh-observability:4317
        - name: KV_URL
          value: http://kv.kong-mesh-demo.svc.cluster.local:5050
        - name: APP_VERSION
          valueFrom:
            fieldRef:
              fieldPath: metadata.labels['version']
        image: ghcr.io/kumahq/kuma-counter-demo:latest@sha256:daf8f5cffa10b576ff845be84e4e3bd5a8a6470c7e66293c5e03a148f08ac148
        name: demo-app
        ports:
        - containerPort: 5050
          name: http
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kv
  namespace: kong-mesh-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kv
  template:
    metadata:
      labels:
        app: kv
    spec:
      containers:
      - env:
        - name: OTEL_SERVICE_NAME
          value: kv
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: http://opentelemetry-collector.mesh-observability:4317
        - name: APP_VERSION
          valueFrom:
            fieldRef:
              fieldPath: metadata.labels['version']
        image: ghcr.io/kumahq/kuma-counter-demo:latest@sha256:daf8f5cffa10b576ff845be84e4e3bd5a8a6470c7e66293c5e03a148f08ac148
        name: app
        ports:
        - containerPort: 5050
          name: http
---
apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  meshServices:
    mode: Exclusive
  mtls:
    backends:
    - name: ca-1
      type: builtin
    enabledBackend: ca-1
---
apiVersion: kuma.io/v1alpha1
kind: MeshTrafficPermission
metadata:
  name: kv
  namespace: kong-mesh-demo
spec:
  from:
  - default:
      action: Allow
    targetRef:
      kind: MeshSubset
      tags:
        app: demo-app
        k8s.kuma.io/namespace: kong-mesh-demo
  targetRef:
    kind: Dataplane
    labels:
      app: kv" | kubectl apply -f -

Deploy a second demo Service

Run the following command to deploy a demo Service in a new kong-mesh-demo-migration namespace outside of the mesh:

echo "
apiVersion: v1
kind: Namespace
metadata:
  labels:
    kuma.io/sidecar-injection: disabled
  name: kong-mesh-demo-migration
---
apiVersion: v1
kind: Service
metadata:
  name: demo-app
  namespace: kong-mesh-demo-migration
spec:
  ports:
  - appProtocol: http
    port: 5050
    protocol: TCP
    targetPort: 5050
  selector:
    app: demo-app
---
apiVersion: v1
kind: Service
metadata:
  name: demo-app-v1
  namespace: kong-mesh-demo-migration
spec:
  ports:
  - appProtocol: http
    port: 5050
    protocol: TCP
    targetPort: 5050
  selector:
    app: demo-app
    version: v1
---
apiVersion: v1
kind: Service
metadata:
  name: demo-app-v2
  namespace: kong-mesh-demo-migration
spec:
  ports:
  - appProtocol: http
    port: 5050
    protocol: TCP
    targetPort: 5050
  selector:
    app: demo-app
    version: v2
---
apiVersion: v1
kind: Service
metadata:
  name: kv
  namespace: kong-mesh-demo-migration
spec:
  ports:
  - appProtocol: http
    port: 5050
    protocol: TCP
    targetPort: 5050
  selector:
    app: kv
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: demo-app
    version: v1
  name: demo-app
  namespace: kong-mesh-demo-migration
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demo-app
      version: v1
  template:
    metadata:
      labels:
        app: demo-app
        version: v1
    spec:
      containers:
      - env:
        - name: OTEL_SERVICE_NAME
          value: demo-app
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: http://opentelemetry-collector.mesh-observability:4317
        - name: KV_URL
          value: http://kv.kong-mesh-demo-migration.svc.cluster.local:5050
        - name: APP_VERSION
          valueFrom:
            fieldRef:
              fieldPath: metadata.labels['version']
        image: ghcr.io/kumahq/kuma-counter-demo:latest@sha256:1472f3e1d7787a2a16af24f5deab7ab7e8316bb6076a8b42f634426d08cad0f8
        name: app
        ports:
        - containerPort: 5050
          name: http
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: demo-app
    version: v2
  name: demo-app-v2
  namespace: kong-mesh-demo-migration
spec:
  replicas: 0
  selector:
    matchLabels:
      app: demo-app
      version: v2
  template:
    metadata:
      labels:
        app: demo-app
        version: v2
    spec:
      containers:
      - env:
        - name: OTEL_SERVICE_NAME
          value: demo-app
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: http://opentelemetry-collector.mesh-observability:4317
        - name: KV_URL
          value: http://kv.kong-mesh-demo-migration.svc.cluster.local:5050
        - name: APP_VERSION
          valueFrom:
            fieldRef:
              fieldPath: metadata.labels['version']
        image: ghcr.io/kumahq/kuma-counter-demo:latest@sha256:1472f3e1d7787a2a16af24f5deab7ab7e8316bb6076a8b42f634426d08cad0f8
        name: demo-app
        ports:
        - containerPort: 5050
          name: http
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kv
  namespace: kong-mesh-demo-migration
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kv
  template:
    metadata:
      labels:
        app: kv
    spec:
      containers:
      - env:
        - name: OTEL_SERVICE_NAME
          value: kv
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: http://opentelemetry-collector.mesh-observability:4317
        - name: APP_VERSION
          valueFrom:
            fieldRef:
              fieldPath: metadata.labels['version']
        image: ghcr.io/kumahq/kuma-counter-demo:latest@sha256:1472f3e1d7787a2a16af24f5deab7ab7e8316bb6076a8b42f634426d08cad0f8
        name: app
        ports:
        - containerPort: 5050
          name: http" | kubectl apply -f -

Allow traffic on the mesh

Run the following command to add a MeshTrafficPermission policy that allows all traffic from inside the mesh:

echo "apiVersion: kuma.io/v1alpha1
kind: MeshTrafficPermission
metadata:
  namespace: kong-mesh-system
  name: mtp
spec:
  targetRef:
    kind: Mesh
  from:
    - targetRef:
        kind: Mesh
      default:
        action: Allow" | kubectl apply -f -

Port-forward the two demo apps

To interact with the two demo-app Pods we created in the prerequisites, we need to enable port-forwarding.

  1. Run the following command to port-forward the first Service: 
    kubectl port-forward svc/demo-app -n kong-mesh-demo 5051:5050

  2. Go to http://localhost:5051/ to access the demo app’s UI and select the Auto Increment checkbox to automatically send requests to the Service.

  3. In a new terminal window, run the following command to port-forward the second Service: 
    kubectl port-forward svc/demo-app -n kong-mesh-demo-migration 5052:5050
  4. Go to http://localhost:5052/ and select the Auto Increment checkbox.

Enable MeshTLS in permissive mode

In a new terminal window, run the following command to enable the MeshTLS policy in permissive mode for the kv app in the kong-mesh-demo-migration namespace:

echo "apiVersion: kuma.io/v1alpha1
kind: MeshTLS
metadata:
  name: kv
  namespace: kong-mesh-demo-migration
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: kv
  rules:
  - default:
      mode: Permissive" | kubectl apply -f -

Add the Pods to the mesh

  1. Run the following command to enable sidecar injection on the kong-mesh-demo-migration namespace to add its Pods to the mesh:

    kubectl label namespace kong-mesh-demo-migration kuma.io/sidecar-injection=enabled --overwrite

  2. Restart the kv Pod to apply the changes:

    kubectl rollout restart deployment kv -n kong-mesh-demo-migration

    This can take a few minutes, make sure to wait until it’s completed to move on to the next step.

Check that the Service is receiving traffic

  1. Once the kv Pod has restarted, enable port-forwarding for the control plane:

    kubectl port-forward svc/kong-mesh-control-plane -n kong-mesh-system 5681:5681
  2. In a new terminal window, export the name of the kv data plane proxies : 
    export KV_DPP_NAME_1=$(curl -s http://localhost:5681/meshes/default/dataplanes/_overview\?name\=kv | jq -c '.items[] | select( .labels["k8s.kuma.io/namespace"]=="kong-mesh-demo")' | jq -r '.name')
export KV_DPP_NAME_2=$(curl -s http://localhost:5681/meshes/default/dataplanes/_overview\?name\=kv | jq -c '.items[] | select( .labels["k8s.kuma.io/namespace"]=="kong-mesh-demo-migration")' | jq -r '.name')

  3. Run the following command to get the request metrics for the data plane proxies:

    for i in {1..2}; do
 sleep 30
 print 'Service kong-mesh-demo'
 curl -s http://localhost:5681/meshes/default/dataplanes/$KV_DPP_NAME_1/stats | grep cluster.localhost_5050.upstream_rq_2xx
 curl -s http://localhost:5681/meshes/default/dataplanes/$KV_DPP_NAME_1/stats | grep http.localhost_5050.rbac.allowed
 print 'Service kong-mesh-demo-migration'
 curl -s http://localhost:5681/meshes/default/dataplanes/$KV_DPP_NAME_2/stats | grep cluster.localhost_5050.upstream_rq_2xx
 curl -s http://localhost:5681/meshes/default/dataplanes/$KV_DPP_NAME_2/stats | grep http.localhost_5050.rbac.allowed
done

    You should get a response similar to this:

    Service kong-mesh-demo
cluster.localhost_5050.upstream_rq_2xx: 871
http.localhost_5050.rbac.allowed: 1310
Service kong-mesh-demo-migration
cluster.localhost_5050.upstream_rq_2xx: 5
http.localhost_5050.rbac.allowed: 0
Service kong-mesh-demo
cluster.localhost_5050.upstream_rq_2xx: 873
http.localhost_5050.rbac.allowed: 1313
Service kong-mesh-demo-migration
cluster.localhost_5050.upstream_rq_2xx: 7
http.localhost_5050.rbac.allowed: 0

    Between the two iterations, you should see:

    • The cluster.localhost_5050.upstream_rq_2xx and http.localhost_5050.rbac.allowed values increase for the data plane proxy in the kong-mesh-demo namespace.
    • The cluster.localhost_5050.upstream_rq_2xx value increase and the http.localhost_5050.rbac.allowed value remain at 0 for the data plane proxy in the kong-mesh-demo-migration namespace.

    This indicates that the proxy in kong-mesh-demo-migration namespace is not receiving encrypted traffic, because the demo-app data plane proxy is not in the mesh.

Migrate the demo app client to the mesh

  1. Run the following command restart the demo-app Pod from the kong-mesh-demo-migration namespace and add it to the mesh:

    kubectl rollout restart deployment demo-app -n kong-mesh-demo-migration
    • This can take a few minutes, make sure to wait until it’s completed to move on to the next step.
    • Once the restart is done, port-forwarding should stop for this Service.
  2. Run the following command to see the status of the migration: 
    kubectl rollout status deployment demo-app -n kong-mesh-demo-migration

    After you see deployment "demo-app" successfully rolled out in the response, you can continue to the next step.

  3. Close the terminal window in which you started the 5052:5050 port-forward to terminate that process.

Validate
  1. Run the following command re-enable port-forwarding: 
    kubectl port-forward svc/demo-app -n kong-mesh-demo-migration 5052:5050

  2. Go to http://localhost:5052/ and select the Auto Increment checkbox to send requests to the Service.

  3. In a new terminal window, run the following command to get the encrypted request metrics for the data plane proxy in the kong-mesh-demo-migration namespace:

    export KV_DPP_NAME_2=$(curl -s http://localhost:5681/meshes/default/dataplanes/_overview\?name\=kv | jq -c '.items[] | select( .labels["k8s.kuma.io/namespace"]=="kong-mesh-demo-migration")' | jq -r '.name')
for i in {1..2}; do
 sleep 30
 curl -s http://localhost:5681/meshes/default/dataplanes/$KV_DPP_NAME_2/stats | grep http.localhost_5050.rbac.allowed
done

    This value should now be increasing:

    http.localhost_5050.rbac.allowed: 1065
http.localhost_5050.rbac.allowed: 1737

FAQs

How do I disable unencrypted traffic after migrating?

To disable unencrypted traffic, update the MeshTLS policy from permissive to strict:

echo "apiVersion: kuma.io/v1alpha1
kind: MeshTLS
metadata:
  name: kv
  namespace: kong-mesh-demo-migration
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: kv
  rules:
  - default:
      mode: Strict" | kubectl apply -f -

The Service can now only receive encrypted traffic.

Something wrong?
Report an Issue | Edit this Page
View all Kong documentation

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!
Contribute

Still need help?

Ask in our Slack Community
Contact Support