Load balancing with Kong Gateway

Every Gateway Service that has been defined with a host containing a hostname automatically uses DNS-based load balancing if the name resolves to multiple IP addresses.

This is similar to round-robin load balancing, but without any health checking functionality.

Advanced load balancing algorithms are available through the Upstream entity. Upstreams let you load-balance traffic across multiple Targets based on a chosen algorithm.

Using a combination of Targets and Upstreams, you can proxy requests to an upstream service through a ring balancer.

The ring balancer is a Kong Gateway load balancer that distributes the traffic load among Targets, and manages active Targets based on their health.

The round-robin algorithm is done in a weighted manner. It provides identical results to the default DNS-based load balancing, and also gives you access to active and passive health checks.

With the consistent-hashing algorithm, a configurable client input is used to calculate a hash value. This hash value is then tied to a specific upstream service.

The least-connections algorithm keeps track of the number of in-flight requests for each upstream service. It uses weights to calculate the connection capacity of an upstream service, and routes requests to the upstream service with the highest spare capacity.

The latency algorithm is based on the peak EWMA (Exponentially Weighted Moving Average), which ensures that the balancer selects the upstream service by the lowest latency.

In round-robin load balancing, AI Gateway maintains a list of target AI models and distributes each request to the next AI model in rotation. The round-robin algorithm is done in a weighted manner.

The consistent-hashing algorithm uses a specified header value as the hash input. It creates a sticky session which ensures that all requests with the same header are dealt with the same way.

The lowest-usage algorithm is based on the volume of usage for each model. It balances the load by distributing requests to models with the lowest usage, measured by factors such as prompt token counts, response token counts, or other resource metrics.

The lowest-latency algorithm is based on the response time for each model. It distributes requests to models with the lowest response time.

The semantic algorithm distributes requests to different models based on the similarity between the prompt in the request and the description provided in the model configuration.

The priority algorithm allows you to define groups of targets with different priority levels and distribute traffic between targets in a group.