Optimize Kong Gateway performance

While Kong Gateway is optimized out-of-the-box, there are still situations where tweaking some configuration options for Kong Gateway can substantially increase its performance.

If you have performed a Kong Gateway benchmark and found performance issues, review the following recommendations to improve your Kong Gateway performance. After making adjustments, perform additional benchmarks.

Always measure, make some changes, and measure again. Maintain a log of changes to help you figure out the next steps when you get stuck or trace back to another approach.

The following recommendations can help you improve Kong Gateway performance.

Action: Increase the ulimit if it’s less than 16384.

Explanation: While Kong Gateway can use as many resources as it can get from the system, the operating system (OS) limits the number of connections Kong Gateway can open with the upstream (or any other) server, or that it can accept from the client.

The number of open connections in Kong Gateway defaults to the ulimit with an upper bound of 16384. This means that if the ulimit is unlimited or is a value higher than 16384, Kong Gateway limits itself to 16384.

You can shell into Kong Gateway’s container or VM and run ulimit -n to check the system’s ulimit. If Kong Gateway is running inside a container on top of a VM, you must shell into the container. If the value of ulimit is less than 16384, increase it.

Also check and set the appropriate ulimit in the client and upstream server, since a connection bottleneck in these systems leads to suboptimal performance.

Action: Set both upstream_keepalive_max_requests and nginx_http_keepalive_requests to 100000.

Explanation: In high throughput scenarios with 10 000 or more RPS, the overhead of setting up TCP and TLS connections or insufficient connections can result in underuse of network bandwidth or the upstream server.

To increase connection reuse, you can increase upstream_keepalive_max_requests and nginx_http_keepalive_requests to 100000, or all the way up to 500000.

Action: Ensure that Kong Gateway is not scaled in/out (horizontal) or up/down (vertical).

Explanation: During a benchmarking run, ensure that Kong Gateway is not scaled in/out (horizontal) or up/down (vertical). In Kubernetes, this is commonly done using a Horizontal or Vertical Pod autoscaler. Autoscalers interfere with statistics in a benchmark and introduce unnecessary noise.

Scale Kong Gateway out before testing the benchmark to avoid autoscaling during the benchmark. Monitor the number of Kong Gateway nodes to ensure new nodes are spawned during the benchmark and existing nodes are not replaced.

Action: On most VM setups, set nginx_worker_processes to auto. On Kubernetes, set nginx_worker_processes to one or two less than the worker node CPUs.

Explanation: Make sure nginx_worker_processes is configured correctly:

• On most VM setups, set this to auto. This is the default setting. This ensures that Nginx spawns one worker process for each CPU core.

• We recommend setting this explicitly in Kubernetes. Ensure that CPU requests and limits for Kong Gateway match the number of workers configured in Kong Gateway. For example, if you configure nginx_worker_processes=4, you must request 4 CPUs in your pod spec.

  If you run Kong Gateway pods on Kubernetes worker nodes with n CPUs, allocate n-2 or n-1 to Kong Gateway, and configure a worker process count equal to this number. This ensures that any configured daemons and Kubernetes processes, like kubelet, don’t contend for resources with Kong Gateway.

  Each additional worker uses additional memory, so you must ensure that Kong Gateway isn’t triggering the Linux Out-of-Memory Killer.

Action: Make sure the client (like Apache JMeter or k6), Kong Gateway, and upstream servers are on different machines (VM or bare metal) and run on the same local network with low latencies.

Explanation:

• Ensure that the client (like Apache JMeter or k6), Kong Gateway, and the upstream servers run on different machines (VM or bare-metal). If these are all running in a Kubernetes cluster, ensure that the pods for these three systems are scheduled on dedicated nodes. Resource contention (usually CPU and network) between these can lead to suboptimal performance of any system.
• Ensure the client, Kong Gateway, and upstream servers run on the same local network with low latencies. If requests between the client and Kong Gateway or Kong Gateway and the upstream server traverse the internet, then the results will contain unnecessary noise.

Action: Verify that the upstream server isn’t maxing out.

Explanation: You can verify that the upstream server isn’t maxing out by checking the CPU and memory usage of the upstream server. If you deploy additional Kong Gateway nodes and the throughput or error rate remains the same, the upstream server or a system other than Kong Gateway is likely the bottleneck.

You must also ensure that upstream servers are not autoscaled.

Action: The client must use keep-alive connections.

Explanation: Sometimes, the clients (such as k6 and Apache JMeter) max themselves out. To tune them, you need to understand the client. Increasing the CPU, threads, and connections on clients results in higher resource use and throughput.

The client must also use keep-alive connections. For example, k6 and the HTTPClient4 implementation in Apache JMeter both enable keep-alive by default. Verify that this is set up appropriately for your test setup.

Action: Ensure that custom plugins aren’t interfering with performance.

Explanation: Custom plugins can sometimes cause issues with performance. First, determine if custom plugins are the source of the performance issues. You can do this by measuring three configuration variations:

1. Measure Kong Gateway’s performance without enabling any plugins. This provides a baseline for Kong Gateway’s performance.
2. Enable necessary bundled plugins (plugins that come with the product), and then measure Kong Gateway’s performance.
3. Next, enable custom plugins (in addition to bundled plugins), and then measure Kong Gateway’s performance once again.

If Kong Gateway’s baseline performance is poor, then it’s likely that either Kong Gateway’s configuration needs tuning or external factors are affecting it. For external factors, see the other sections in this guide. A large difference between the performance in the second and third steps indicates that performance problems could be due to custom plugins.

Action: Ensure you aren’t using burstable instances or hitting bandwidth, TCP connection per unit time, or PPS limits.

Explanation: While AWS is mentioned in the following, the same recommendations apply to most cloud providers:

• Ensure that you aren’t using burstable instances, like T type instances, in AWS. In this case, the CPU available to applications is variable, which leads to noise in the stats. For more information, see the Burstable performance instances AWS documentation.
• Ensure you aren’t hitting bandwidth limits, TCP connections per unit time limits, or Packet Per Second (PPS) limits. For more information, see the Amazon EC2 instance network bandwidth AWS documentation.

Action: Don’t change Kong Gateway configuration during a benchmark test.

Explanation: If you change the configuration during a test, Kong Gateway’s tail latencies can increase sharply. Avoid doing this unless you are measuring Kong Gateway’s performance under a configuration change.

Action: Keep request bodies below 8 KB and response bodies below 32 KB.

Explanation: Most benchmarking setups generally consist of an HTTP request with a small HTTP body and a corresponding HTTP response with a JSON or HTML response body. A request body of less than 8 KB and a response body of less than 32 KB is considered small. If your request or response bodies are larger, Kong Gateway will buffer the request and response using the disk, which significantly impacts Kong Gateway’s performance.

More often than not, the bottlenecks in Kong Gateway are caused by bottlenecks in third-party systems used by Kong Gateway. The following sections explain common third-party bottlenecks and how to fix them.

Redis

Action: If you use Redis and any plugin is enabled, the CPU can cause a bottleneck. Scale Redis vertically by giving it an additional CPU.

Explanation: If you use Redis and any plugin is enabled, ensure Redis is not a bottleneck. The CPU generally creates a bottleneck for Redis, so check CPU usage first. If this is the case, scale Redis vertically by giving it an additional CPU.

DNS client v3.8+

Action: Migrate to the new DNS client.

Explanation: The new DNS client is designed to be more performant than the old one, so migrating will improve performance. For more information, see the migration docs.

DNS TTL

Action: Increase dns_stale_ttl or resolver_stale_ttl v3.8+ to 300 or up to 86400.

Explanation: DNS servers can bottleneck Kong Gateway since Kong Gateway depends on DNS to determine where to send the request.

In the case of Kubernetes, DNS TTLs are 5 seconds long and can cause problems. You can increase dns_stale_ttl or resolver_stale_ttl, depending on the Kong Gateway version and DNS client you are using, to 300 or up to 86400 to rule out DNS as the issue.

If DNS servers are the root cause, you will see coredns pods creating a bottleneck on the CPU.

Action: Disable access logs for high throughput benchmarking tests by setting the proxy_access_log configuration parameter to off.

Explanation: Kong Gateway and the underlying Nginx are programmed for non-blocking network I/O and they avoid blocking disk I/O as much as possible. However, access logs are enabled by default, and if the disk powering a Kong Gateway node is slow for any reason, it can result in performance loss. Disable access logs for high throughput benchmarking tests by setting the proxy_access_log configuration parameter to off.

Action: Make sure that there are no errors in Kong Gateway’s error log.

Explanation: Check Kong Gateway’s error log for internal errors. Internal errors can highlight issues within Kong Gateway or a third-party system that Kong Gateway relies on to proxy traffic.

The following kong.conf file examples contain all the recommended parameters from the previous sections:

# For a Kubernetes setup, change nginx_worker_processes to a number matching the CPU limit. We recommend 4 or 8.
nginx_worker_processes=auto

upstream_keepalive_max_requests=100000
nginx_http_keepalive_requests=100000

proxy_access_log=off

dns_stale_ttl=3600

# For a Kubernetes setup, change nginx_worker_processes to a number matching the CPU limit. We recommend 4 or 8.
KONG_NGINX_WORKER_PROCESSES="auto"
KONG_UPSTREAM_KEEPALIVE_MAX_REQUESTS="100000"
KONG_NGINX_HTTP_KEEPALIVE_REQUESTS="100000"

KONG_PROXY_ACCESS_LOG="off"

KONG_DNS_STALE_TTL="3600"

# The value of 1 for nginx_worker_processes is a suggested value. 
# Change nginx_worker_processes to a number matching the CPU limit. We recommend 4 or 8.
# Allocate the same amount of CPU and appropriate memory to avoid OOM killer.
env:
  nginx_worker_processes: "1"
  upstream_keepalive_max_requests: "100000"
  nginx_http_keepalive_requests: "100000"
  proxy_access_log: "off"
  dns_stale_ttl: "3600"

resources:
  requests:
    cpu: 1
    memory: "2Gi"