Upgrading Kong Ingress Controller using Helm

If you are upgrading from Kong Ingress Controller 2.12, read the upgrading to 3.0 section first

1. Ensure that you installed Kong Ingress Controller 3.0 or above, using Helm.

2. Fetch the latest version of the Kong Helm chart using helm repo update.

3. Update your values.yaml file to use the latest version of Kong Ingress Controller. The values to set are different depending on if you’ve installed with the kong/ingress chart or the kong/kong chart. You can find which chart you’re using by running helm list -A -o yaml | grep chart.

   kong/ingress kong/kong

   controller:
     ingressController:
       image:
         tag: 3.4

   ingressController:
     image:
       tag: 3.4

Custom resource definitions (CRDs) are not upgraded when running helm upgrade. To manually upgrade the CRDs in your cluster, run the following command:

kubectl kustomize https://github.com/kong/kubernetes-ingress-controller/config/crd?ref=v3.4 | kubectl apply -f -

Gateway API resources are constantly evolving, and Kong Ingress Controller is updated to use the latest version of the Gateway API CRDs with each release.

Kong Ingress Controller 3.2 and 3.4 contain Gateway API changes that require operator intervention.

Kong Ingress Controller 3.2

Starting from version 3.2, Kong Ingress Controller supports Gateway API version 1.1. The primary change in Gateway API v1.1 is the promotion of GRPCRoute from v1alpha2 to v1.

If you have been using the Standard Channel of the Gateway API, then you don’t need to do anything extra. Download the latest version of the CRD from the Standard Channel and install it in your cluster directly.

If you installed the Experimental Channel of Gateway API v1.0, complete the following steps to upgrade to version v1.1 of the CRD.

1. Install the Experimental Channel of Gateway API v1.1:

   kubectl apply --force=true -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.1.0/experimental-install.yaml  
   

   --force=true is necessary because upstream updated the CRD directly for the alpha stage without preserving the definition of the old version.

2. Update all your manifests to use v1 instead of v1alpha2.
3. Upgrade to Kong Ingress Controller v3.2.

Kong Ingress Controller 3.4

Starting in Kong Ingress Controller 3.4, Kong Ingress Controller supports Gateway API version 1.2. There is a breaking change in Gateway API 1.2 to remove the v1alpha2 version of GRPCRoute and ReferenceGrant.

If you have been using Gateway API v1.1 and Kong Ingress Controller 3.2 and above, and there are no GRPCRoute and ReferenceGrant resources stored in v1alpha2 version, you can directly upgrade Gateway API from v1.1 to v1.2. You can use the following script to ensure your GRPCRoute and ReferenceGrant CRDs aren’t using v1alpha2 storage version:

kubectl get grpcroutes -A -o jsonpath='{.items[*].apiVersion}' | tr ' ' '\n' | sort | uniq -c
kubectl get referencegrants -A -o jsonpath='{.items[*].apiVersion}' | tr ' ' '\n' | sort | uniq -c

If the output contains v1alpha2, it means that there are one or more GRPCRoute or ReferenceGrant (or both) using v1alpha2 storage version and you must update the manifests before upgrading.

Otherwise, upgrade Gateway API and Kong Ingress Controller following these steps:

1. Ensure you are using Gateway API 1.1 (you can upgrade by following the steps in the section above).
2. Ensure you are using Kong Ingress Controller version 3.2 or later.
3. Update all your GRPCRoute manifests to use v1 instead of v1alpha2 and your ReferenceGrant manifests to use v1beta1 instead of v1alpha2. This can be done by following the upgrade guide from Gateway API.
4. Install the standard or experimental channel of Gateway API 1.2.

Run the following command, specifying the old release name, the namespace where you’ve configured Kong Gateway, and the existing values.yaml configuration file.

 helm upgrade $YOUR_RELEASE_NAME kong/ingress \
  --namespace $YOUR_NAMESPACE \
  -f ./values.yaml

 helm upgrade $YOUR_RELEASE_NAME kong/kong \
  --namespace $YOUR_NAMESPACE \
  -f ./values.yaml

After the upgrade completes, there’s a brief period of time before the new resources are online. You can wait for the relevant Pod resources to cycle by watching them in your release namespace:

 kubectl -n $YOUR_RELEASE_NAMESPACE get pods -w

Once the new pods are in a Ready state, the upgrade is complete.

If you run into problems during or after the upgrade, Helm provides a rollback mechanism to revert to a previous revision of the release.

 helm rollback --namespace $YOUR_RELEASE_NAMESPACE $YOUR_RELEASE_NAME

You can wait for the rollback to complete by watching the relevant Pod resources:

 kubectl -n $YOUR_RELEASE_NAMESPACE get pods -w

After a rollback, if you run into issues in production, consider using a testing environment to identify and correct these issues, or reference the troubleshooting documentation.

Upgrading from Kong Ingress Controller 2.12 to 3.x+ is a major version change which contains breaking changes. To safely upgrade, follow these steps.

1. Switch to Helm as your deployment mechanism.

   As of Kong Ingress Controller 3.0, Helm is the only officially supported install method.

2. Upgrade Kong to version 3.4.1 or later.

   Kong Ingress Controller 3.0 requires Kong Gateway 3.4.1 or later. You must upgrade your Kong Gateway instances to 3.4.1 before you upgrade to Kong Ingress Controller 3.0.

3. Update the Kong Ingress Controller CRDs.

   Helm does not upgrade CRDs automatically. You must apply the 3.x CRDs before you upgrade your releases.

    kubectl kustomize https://github.com/Kong/kubernetes-ingress-controller/config/crd/?ref=v3.0.0 | kubectl apply -f -
   

4. Convert KongIngress route and service fields to annotations.

   Route (Ingress) and Service (Service) configuration fields previously available in KongIngress are now all handled via dedicated annotations and will not be respected if set in KongIngress.

   For example, if you set the route.https_redirect_status_code in a KongIngress resource, you should now use the konghq.com/https-redirect-status-code annotation on an Ingress or HTTPRoute resource.

5. Remove the CombinedRoutes and CombinedServices feature gates if set.

   The CombinedRoutes and CombinedServices feature gates have been enabled by default since versions 2.8.0 and 2.11.0, respectively. Version 3.x removes these feature gates and the combined generators are now the only option. You must remove these flags from the CONTROLLER_FEATURE_GATES environment variable if they are present.

6. Remove the Knative feature gate if set.

   As KNative is no longer supported, you need to use another controller for KNative Ingress resources if you use them.

7. Remove or rename outdated CLI arguments and CONTROLLER_* environment variables.

   Version 3.0 removes or renames several flags that were previously deprecated, were removed due to other changes, or were left over for compatibility after their functionality was removed.

   The CLI argument versions of flags are listed below. You must also change the equivalent CONTROLLER_ (for example, CONTROLLER_SYNC_RATE_LIMIT for --sync-rate-limit) environment variable if you use those.

   • --sync-rate-limit is now --proxy-sync-seconds.
   • --konnect-runtime-group-id is now --konnect-control-plane-id.
   • --stderrthreshold and --debug-log-reduce-redundancy were removed following changes to the logging system.
   • --log-level no longer accepts the warn, fatal, and panic values due to consolidation of log levels.
   • --update-status-on-shutdown was removed after its earlier functionality was removed.
   • --kong-custom-entities-secret was removed after its functionality was removed in 2.0.
   • --leader-elect was removed. The controller automatically configures its leader election mode based on other settings.
   • --enable-controller-ingress-extensionsv1beta1 and --enable-controller-ingress-networkingv1beta1 were removed following removal of support for older Ingress API versions.

Notable changes

The following changes are not considered breaking changes. However, they are notable changes from the previous version and are documented here for completeness.

Expression Router

Kong Gateway 3.0 introduced a new expression-based routing engine. This engine allows Kong Ingress Controller to set some match criteria and route matching precedence not possible under the original Kong Gateway routing engine. This functionality is necessary to implement some aspects of the Gateway API specification.

DB-less configurations in the Helm chart now use the expressions router_flavor kong.conf setting by default to take advantage of this functionality. DB-backed configurations use traditional_compatible instead for backwards compatibility, as existing route configuration from older versions cannot yet be migrated in DB mode.

Use of the new routing engine should not change route matching outside of cases where route precedence did not match the Gateway API specification. The new engine does have different performance characteristics than the old engine, but should improve matching and configuration update speed for most configurations.

Logging changes

Kong Ingress Controller 3.0 uses a new logging engine to unify the logging implementation across all its subsystems. Earlier versions used different logging engines in different subsystems, which led to some logs using a different format than others and some logs appearing at incorrect log levels.

The new logging system consolidates log levels into debug, info, and error. Logs that were previously logged at the warn level are now logged at error, as the conditions that triggered warn level logs were infrequent and should not occur under normal circumstances. fatal and panic levels were available in configuration, but were not actually used.

The new logging system changes the default console format. In earlier versions, console logs used a key=value format:

time="2023-09-21T23:07:26Z" level=info msg="the ingress class name has been set" logger=setup value=kong othervalue=pong

In 3.0, console is a mixture of unlabeled tab-delimited fields (for standard keys such as timestamp, log level, and log section) and JSON (for fields specific to individual log entries):

2023-09-22T00:38:16.026Z        info    setup   the ingress class name has been set     {"value": "kong","othervalue":"pong"}

The json format is unchanged except for the order of fields. Earlier versions printed fields in alphabetical order:

{"level":"info","logger":"setup","msg":"the ingress class name has been set","time":"2023-09-21T23:15:15Z","value":"kong"}

Kong Ingress Controller 3.0 or later prints standard log fields first and entry-specific fields in the order they were added in code:

{"level":"info","time":"2023-09-22T00:28:13.006Z","logger":"setup","msg":"the ingress class name has been set","value":"kong"}

Although the default log setting is still console, json should be used for production systems, or any other systems that need machine-parseable logs.