Incremental configuration sync

In Hybrid mode, whenever you make changes to Kong Gateway entity configuration on the Control Plane, it immediately triggers a cluster-wide update of all Data Plane configurations. In these updates, Kong Gateway sends the entire configuration set to the Data Planes. The bigger your configuration set is, the more time it takes to send and process, and the more memory is consumed proportional to the configuration size. This can result in latency spikes and loss in throughput for high-traffic Data Planes under certain conditions.

You can enable incremental configuration sync to address this issue. When entity configuration changes, instead of sending the entire configuration set for each change, Kong Gateway only sends the parts of the configuration that have changed.

 
flowchart TD

A[Client]
B( Kong Control Plane)
C( Kong Data Plane)
D( Kong Data Plane)
E[Client]
F( Kong Control Plane)
G( Kong Data Plane)
H( Kong Data Plane)

 subgraph id1 [Incremental config sync]
 direction TB

 E --"POST Route config
 1 entity
 A few KB"---> F --"Updated Route config
 1 entity
 A few KB"---> G & H
 end

 subgraph id2 [No incremental config sync]
 direction TB

 A --"POST Route config
 1 entity
 A few KB"---> B --Full Kong config
 30k entities
 30MB---> C & D

 end

style id1 stroke-dasharray:3,rx:10,ry:10
style id2 stroke-dasharray:3,rx:10,ry:10

  

Figure 1: In an environment with 30k entities of about 30MB total, sending a POST request to update one entity sends the whole 30MB config to every Data Plane. With incremental config sync enabled, that same POST request only triggers an update of a few KB.

Incremental config sync achieves significant memory savings and CPU savings. This means lower total cost of ownership for Kong Gateway users, shorter config propagation delay, and less impact to proxy latency. See our blog on incremental config sync for the performance comparisons.

You can enable incremental config sync when installing Kong Gateway in hybrid mode, or when setting up a Konnect Data Plane.

Caution: There are some limitations for custom plugins and for rate limiting plugins when using incremental config sync. Review and adjust your plugin config before enabling this feature.

During setup, set the following value in your kong.conf files on both Control Planes and Data Planes:

incremental_sync = on

Or, if you’re running Kong Gateway in Docker, set the following environment variable:

export KONG_INCREMENTAL_SYNC=on

When using incremental config sync feature with plugins, you may encounter the following limitations.

Using incremental config sync with custom plugins

When incremental config sync is enabled, the configuration change notification from the Control Plane only triggers an event for changed entities, and doesn’t trigger cache updates in Data Plane nodes. This causes outdated and inconsistent configuration for custom plugins.

If you are running Kong Gateway on Konnect or in hybrid mode, you need to adjust your custom plugins to be compatible with incremental config sync.

Workaround for custom plugins

To ensure your custom plugin configuration is kept up to date, you must add additional code logic to register the CRUD events for the entities the plugin cares about, and invalidate the relevant cache data.

Add the code at the end of the init_worker function for the custom plugin. For example:

function _M.init_worker()
  -- ...
  -- ... your original custom code
  -- add the example code below AFTER any custom plugin code
 
  -- Check if worker_events can be registed successfully
  if not (kong.worker_events and kong.worker_events.register) then
    return
  end

  -- Check the deployment mode and incremental sync feature is enabled
  if kong.configuration.database == "off" and not kong.sync then
    return
  end
  
  -- Do the event registration and force invalidate the corresponding cache
  kong.worker_events.register(
    function(data)
    -- logic to identify what cache need to be invalidated
    kong.cache:invalidate(CACHE_KEY) -- Your plugin logic to invalidate the cache
  end, "crud", "ENTITY_NAME") -- Register the events for entity the plugin cares
  
  -- Repeat the register events for other ENTITIES
  
end

Using incremental config sync with rate limiting plugins

We don’t recommend using the local strategy for rate limiting plugins with incremental config sync.

When load balancing across multiple Data Plane nodes, rate limiting is enforced per node. With the local strategy, rapid configuration updates may cause inconsistencies and potential resets in rate limiting plugins, impacting performance for API traffic control.