Home / Kong Mesh
Edit
Report

Configure the Kong Mesh CNI

Uses: Kong Mesh
Related Documentation
Mesh Documentation
Minimum Version
Kong Mesh - 2.9
Tags
#kubernetes #network
Related Resources
Mesh DNS
Transparent proxying
Kubernetes annotations and labels

For traffic to flow through Kong Mesh, all inbound and outbound traffic for a service must pass through its sidecar proxy. The recommended way of accomplishing this is via transparent proxying.

On Kubernetes, this is handled automatically by the kuma-init container, which requires elevated privileges for every Pod in the mesh. The Kong Mesh CNI is an alternative that removes this requirement, which can make security compliance easier.

The CNI DaemonSet itself requires a privileged container security context (containerSecurityContext.privileged=true) because it writes executables to the host filesystem as root.

You can install the CNI using either kumactl or Helm. The default settings are optimized for OpenShift with Multus. To use Kong Mesh CNI in other environments, follow the installation instructions below.

Kong Mesh CNI applies NetworkAttachmentDefinitions to applications in any namespace with kuma.io/sidecar-injection label. To apply NetworkAttachmentDefinitions to applications not in a mesh, add the label kuma.io/sidecar-injection with the value disabled to the namespace.

CNI architecture

The CNI DaemonSet kong-mesh-cni consists of two components:

  • A CNI installer
  • A CNI binary

The components interact as follows:

 
flowchart LR
 subgraph s1["conflist"]
        n2["existing CNIs"]
        n3["kong-mesh-cni"]
  end
 subgraph s2["application pod"]
        n4["kuma-sidecar"]
        n5["app-container"]
  end
    A["installer"] -- copy binary and setup conf --> n3
    n3 -- configure iptables --> n4
  1. The CNI installer copies the kong-mesh-cni binary to the CNI directory on the host. When CNI chaining is enabled, it also sets up chaining for kong-mesh-cni in the CNI conflist file. When chaining is disabled, kong-mesh-cni is invoked explicitly as per the Pod manifest.
  2. When a mesh-enabled application Pod is created, Kubernetes invokes the kong-mesh-cni binary, which configures the iptables rules required by the kuma-sidecar container.
  3. When chained, if the CNI conflist file is unexpectedly changed and kong-mesh-cni is excluded, the installer immediately detects it and restarts itself to re-run the chaining installation and restore CNI functionality.

Installation

The following sections contain CNI installation instructions for different environments.

AWS - EKS

Use the following settings to install Kong Mesh CNI on Amazon EKS.

kumactl install control-plane \
  --set "kuma.cni.enabled=true" \
  --set "kuma.cni.chained=true" \
  --set "kuma.cni.netDir=/etc/cni/net.d" \
  --set "kuma.cni.binDir=/opt/cni/bin" \
  --set "kuma.cni.confName=10-aws.conflist" \
  --set "kuma.controlPlane.envVars.KUMA_RUNTIME_KUBERNETES_INJECTOR_SIDECAR_CONTAINER_IP_FAMILY_MODE=ipv4" \
  | kubectl apply -f -

Add KUMA_RUNTIME_KUBERNETES_INJECTOR_SIDECAR_CONTAINER_IP_FAMILY_MODE=ipv4 as EKS has IPv6 disabled by default.

Azure

Use the following settings to install Kong Mesh CNI on Azure Kubernetes Service (AKS).

kumactl install control-plane \
  --set "kuma.cni.enabled=true" \
  --set "kuma.cni.chained=true" \
  --set "kuma.cni.netDir=/etc/cni/net.d" \
  --set "kuma.cni.binDir=/opt/cni/bin" \
  --set "kuma.cni.confName=10-azure.conflist" \
  | kubectl apply -f -

Azure Overlay

Use the following settings to install Kong Mesh CNI on AKS with Azure CNI Overlay networking.

kumactl install control-plane \
  --set "kuma.cni.enabled=true" \
  --set "kuma.cni.chained=true" \
  --set "kuma.cni.netDir=/etc/cni/net.d" \
  --set "kuma.cni.binDir=/opt/cni/bin" \
  --set "kuma.cni.confName=15-azure-swift-overlay.conflist" \
  | kubectl apply -f -

Calico

Use the following settings to install Kong Mesh CNI in a Calico-managed cluster:

kumactl install control-plane \
  --set "kuma.cni.enabled=true" \
  --set "kuma.cni.chained=true" \
  --set "kuma.cni.netDir=/etc/cni/net.d" \
  --set "kuma.cni.binDir=/opt/cni/bin" \
  --set "kuma.cni.confName=10-calico.conflist" \
  | kubectl apply -f -

To install the Kong Mesh CNI with Calico on GKE, follow the Google - GKE section.

Cilium

Use the following settings to install Kong Mesh CNI in a Cilium-managed cluster:

kumactl install control-plane \
  --set "kuma.cni.enabled=true" \
  --set "kuma.cni.chained=true" \
  --set "kuma.cni.netDir=/etc/cni/net.d" \
  --set "kuma.cni.binDir=/opt/cni/bin" \
  --set "kuma.cni.confName=05-cilium.conflist" \
  | kubectl apply -f -
  • You must set the Cilium configuration value cni-exclusive or the corresponding Helm chart value cni.exclusive to false in order to use Cilium and Kong Mesh together. This is necessary starting with the Cilium v1.14.
  • For Cilium versions older than 1.14, use kuma.cni.confName=05-cilium.conf instead of kuma.cni.confName=05-cilium.conflist.

To install the Kong Mesh CNI with Cilium on GKE, follow the Google - GKE section.

Google - GKE

To install the Kong Mesh CNI on GKE, enable network-policy in your cluster first (for existing clusters, this redeploys the nodes).

Define the variable CNI_CONF_NAME for your CNI, for example:

  • export CNI_CONF_NAME=05-cilium.conflist for Cilium
  • export CNI_CONF_NAME=10-calico.conflist for GKE Dataplane V1
  • export CNI_CONF_NAME=10-gke-ptp.conflist for GKE Dataplane V2
kumactl install control-plane \
  --set "kuma.cni.enabled=true" \
  --set "kuma.cni.chained=true" \
  --set "kuma.cni.netDir=/etc/cni/net.d" \
  --set "kuma.cni.binDir=/home/kubernetes/bin" \
  --set "kuma.cni.confName=${CNI_CONF_NAME}" \
  | kubectl apply -f -

K3D with Flannel

Use the following settings to install Kong Mesh CNI on K3D with Flannel:

kumactl install control-plane \
  --set "kuma.cni.enabled=true" \
  --set "kuma.cni.chained=true" \
  --set "kuma.cni.netDir=/var/lib/rancher/k3s/agent/etc/cni/net.d" \
  --set "kuma.cni.binDir=/bin" \
  --set "kuma.cni.confName=10-flannel.conflist" \
  | kubectl apply -f -

Kind

Use the following settings to install Kong Mesh CNI on a Kind cluster:

kumactl install control-plane \
  --set "kuma.cni.enabled=true" \
  --set "kuma.cni.chained=true" \
  --set "kuma.cni.netDir=/etc/cni/net.d" \
  --set "kuma.cni.binDir=/opt/cni/bin" \
  --set "kuma.cni.confName=10-kindnet.conflist" \
  | kubectl apply -f -

OpenShift 3.11

To install the Kong Mesh CNI on OpenShift 3.11, configure admission webhooks and grant the CNI service account the required privileges:

  1. Install OpenShift 3.11 and enable MutatingAdmissionWebhook and ValidatingAdmissionWebhook.

  2. Grant privileged permissions to the kong-mesh-cni service account:

    oc adm policy add-scc-to-user privileged -z kong-mesh-cni -n kube-system

  3. Install the CNI:

    kumactl install control-plane \
  --set "kuma.cni.enabled=true" \
  --set "kuma.cni.containerSecurityContext.privileged=true" \
  | kubectl apply -f -

OpenShift 4

Use the following settings to install Kong Mesh CNI on OpenShift 4:

kumactl install control-plane \
  --set "kuma.cni.enabled=true" \
  --set "kuma.cni.containerSecurityContext.privileged=true" \
  | kubectl apply -f -

CNI taint controller

To prevent a race condition, the taint controller taints new nodes with NoSchedule until the CNI DaemonSet is running and ready, then removes the taint to allow Pod scheduling.

To disable the taint controller, use the following env variable:

KUMA_RUNTIME_KUBERNETES_NODE_TAINT_CONTROLLER_ENABLED="false"

Merbridge CNI with eBPF

To use Merbridge CNI with eBPF, your environment must use kernel ≥ 5.7 and have cgroup2 available.

To install Merbridge CNI with eBPF, append the following options to your install command:

--set "kuma.cni.enabled=true" \
--set "kuma.experimental.ebpf.enabled=true"

CNI logs

CNI component logs are available via kubectl logs.

To enable debug-level logging, set the CNI_LOG_LEVEL environment variable to debug on the kong-mesh-cni DaemonSet. Note that editing the DaemonSet restarts the CNI Pods, during which mesh-enabled application Pods can’t start or stop. Avoid this in production unless approved.

The eBPF CNI currently doesn’t support exposing its logs.

Something wrong?
Report an Issue | Edit this Page
View all Kong documentation

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!
Contribute

Still need help?

Ask in our Slack Community
Contact Support