AI Rate Limiting Advanced: Enable LLM provider rate limiting - Plugin

Enable LLM provider rate limitingv3.7+

Protect your LLM services with rate limiting. The AI Rate Limiting Advanced plugin will analyze query costs and token response to provide an enterprise-grade rate limiting strategy.

The following example uses OpenAI, but you can apply the same strategies to any supported LLM provider.

Prerequisites

AI Proxy plugin or AI Proxy Advanced plugin configured with an LLM service

Set up the plugin

Add this section to your declarative configuration file:

_format_version: "3.0"
plugins:
  - name: ai-rate-limiting-advanced
    config:
      llm_providers:
      - name: openai
        limit:
        - 100
        - 1000
        window_size:
        - 60
        - 3600

Make the following request:

curl -i -X POST http://localhost:8001/plugins/ \
    --header "Accept: application/json" \
    --header "Content-Type: application/json" \
    --data '
    {
      "name": "ai-rate-limiting-advanced",
      "config": {
        "llm_providers": [
          {
            "name": "openai",
            "limit": [
              100,
              1000
            ],
            "window_size": [
              60,
              3600
            ]
          }
        ]
      }
    }
    '

Make the following request:

curl -X POST https://{region}.api.konghq.com/v2/control-planes/{controlPlaneId}/core-entities/plugins/ \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "name": "ai-rate-limiting-advanced",
      "config": {
        "llm_providers": [
          {
            "name": "openai",
            "limit": [
              100,
              1000
            ],
            "window_size": [
              60,
              3600
            ]
          }
        ]
      }
    }
    '

Make sure to replace the following placeholders with your own values:

region: Geographic region where your Kong Konnect is hosted and operates.
controlPlaneId: The id of the control plane.
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account.

See the Konnect API reference to learn about region-specific URLs and personal access tokens.

echo "
apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  name: ai-rate-limiting-advanced
  namespace: kong
  annotations:
    kubernetes.io/ingress.class: kong
  labels:
    global: 'true'
config:
  llm_providers:
  - name: openai
    limit:
    - 100
    - 1000
    window_size:
    - 60
    - 3600
plugin: ai-rate-limiting-advanced
" | kubectl apply -f -

Prerequisite: Configure your Personal Access Token

terraform {
  required_providers {
    konnect = {
      source  = "kong/konnect"
    }
  }
}

provider "konnect" {
  personal_access_token = "$KONNECT_TOKEN"
  server_url            = "https://us.api.konghq.com/"
}

Add the following to your Terraform configuration to create a Konnect Gateway Plugin:

resource "konnect_gateway_plugin_ai_rate_limiting_advanced" "my_ai_rate_limiting_advanced" {
  enabled = true

  config = {
    llm_providers = [
      {
        name = "openai"
        limit = [100, 1000]
        window_size = [60, 3600]
      }    ]
  }

  control_plane_id = konnect_gateway_control_plane.my_konnect_cp.id
}

Add this section to your declarative configuration file:

_format_version: "3.0"
plugins:
  - name: ai-rate-limiting-advanced
    service: serviceName|Id
    config:
      llm_providers:
      - name: openai
        limit:
        - 100
        - 1000
        window_size:
        - 60
        - 3600

Make sure to replace the following placeholders with your own values:

serviceName|Id: The id or name of the service the plugin configuration will target.

Make the following request:

curl -i -X POST http://localhost:8001/services/{serviceName|Id}/plugins/ \
    --header "Accept: application/json" \
    --header "Content-Type: application/json" \
    --data '
    {
      "name": "ai-rate-limiting-advanced",
      "config": {
        "llm_providers": [
          {
            "name": "openai",
            "limit": [
              100,
              1000
            ],
            "window_size": [
              60,
              3600
            ]
          }
        ]
      }
    }
    '

Make sure to replace the following placeholders with your own values:

serviceName|Id: The id or name of the service the plugin configuration will target.

Make the following request:

curl -X POST https://{region}.api.konghq.com/v2/control-planes/{controlPlaneId}/core-entities/services/{serviceId}/plugins/ \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "name": "ai-rate-limiting-advanced",
      "config": {
        "llm_providers": [
          {
            "name": "openai",
            "limit": [
              100,
              1000
            ],
            "window_size": [
              60,
              3600
            ]
          }
        ]
      }
    }
    '

Make sure to replace the following placeholders with your own values:

region: Geographic region where your Kong Konnect is hosted and operates.
controlPlaneId: The id of the control plane.
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account.
serviceId: The id of the service the plugin configuration will target.

See the Konnect API reference to learn about region-specific URLs and personal access tokens.

echo "
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: ai-rate-limiting-advanced
  namespace: kong
  annotations:
    kubernetes.io/ingress.class: kong
config:
  llm_providers:
  - name: openai
    limit:
    - 100
    - 1000
    window_size:
    - 60
    - 3600
plugin: ai-rate-limiting-advanced
" | kubectl apply -f -

Next, apply the KongPlugin resource by annotating the service resource:

kubectl annotate -n kong service SERVICE_NAME konghq.com/plugins=ai-rate-limiting-advanced

Prerequisite: Configure your Personal Access Token

terraform {
  required_providers {
    konnect = {
      source  = "kong/konnect"
    }
  }
}

provider "konnect" {
  personal_access_token = "$KONNECT_TOKEN"
  server_url            = "https://us.api.konghq.com/"
}

Add the following to your Terraform configuration to create a Konnect Gateway Plugin:

resource "konnect_gateway_plugin_ai_rate_limiting_advanced" "my_ai_rate_limiting_advanced" {
  enabled = true

  config = {
    llm_providers = [
      {
        name = "openai"
        limit = [100, 1000]
        window_size = [60, 3600]
      }    ]
  }

  control_plane_id = konnect_gateway_control_plane.my_konnect_cp.id
  service = {
    id = konnect_gateway_service.my_service.id
  }
}

Add this section to your declarative configuration file:

_format_version: "3.0"
plugins:
  - name: ai-rate-limiting-advanced
    route: routeName|Id
    config:
      llm_providers:
      - name: openai
        limit:
        - 100
        - 1000
        window_size:
        - 60
        - 3600

Make sure to replace the following placeholders with your own values:

routeName|Id: The id or name of the route the plugin configuration will target.

Make the following request:

curl -i -X POST http://localhost:8001/routes/{routeName|Id}/plugins/ \
    --header "Accept: application/json" \
    --header "Content-Type: application/json" \
    --data '
    {
      "name": "ai-rate-limiting-advanced",
      "config": {
        "llm_providers": [
          {
            "name": "openai",
            "limit": [
              100,
              1000
            ],
            "window_size": [
              60,
              3600
            ]
          }
        ]
      }
    }
    '

Make sure to replace the following placeholders with your own values:

routeName|Id: The id or name of the route the plugin configuration will target.

Make the following request:

curl -X POST https://{region}.api.konghq.com/v2/control-planes/{controlPlaneId}/core-entities/routes/{routeId}/plugins/ \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "name": "ai-rate-limiting-advanced",
      "config": {
        "llm_providers": [
          {
            "name": "openai",
            "limit": [
              100,
              1000
            ],
            "window_size": [
              60,
              3600
            ]
          }
        ]
      }
    }
    '

Make sure to replace the following placeholders with your own values:

region: Geographic region where your Kong Konnect is hosted and operates.
controlPlaneId: The id of the control plane.
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account.
routeId: The id of the route the plugin configuration will target.

See the Konnect API reference to learn about region-specific URLs and personal access tokens.

echo "
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: ai-rate-limiting-advanced
  namespace: kong
  annotations:
    kubernetes.io/ingress.class: kong
config:
  llm_providers:
  - name: openai
    limit:
    - 100
    - 1000
    window_size:
    - 60
    - 3600
plugin: ai-rate-limiting-advanced
" | kubectl apply -f -

Next, apply the KongPlugin resource by annotating the httproute or ingress resource:

kubectl annotate -n kong httproute  konghq.com/plugins=ai-rate-limiting-advanced

kubectl annotate -n kong ingress  konghq.com/plugins=ai-rate-limiting-advanced

Prerequisite: Configure your Personal Access Token

terraform {
  required_providers {
    konnect = {
      source  = "kong/konnect"
    }
  }
}

provider "konnect" {
  personal_access_token = "$KONNECT_TOKEN"
  server_url            = "https://us.api.konghq.com/"
}

Add the following to your Terraform configuration to create a Konnect Gateway Plugin:

resource "konnect_gateway_plugin_ai_rate_limiting_advanced" "my_ai_rate_limiting_advanced" {
  enabled = true

  config = {
    llm_providers = [
      {
        name = "openai"
        limit = [100, 1000]
        window_size = [60, 3600]
      }    ]
  }

  control_plane_id = konnect_gateway_control_plane.my_konnect_cp.id
  route = {
    id = konnect_gateway_route.my_route.id
  }
}

Add this section to your declarative configuration file:

_format_version: "3.0"
plugins:
  - name: ai-rate-limiting-advanced
    consumer: consumerName|Id
    config:
      llm_providers:
      - name: openai
        limit:
        - 100
        - 1000
        window_size:
        - 60
        - 3600

Make sure to replace the following placeholders with your own values:

consumerName|Id: The id or name of the consumer the plugin configuration will target.

Make the following request:

curl -i -X POST http://localhost:8001/consumers/{consumerName|Id}/plugins/ \
    --header "Accept: application/json" \
    --header "Content-Type: application/json" \
    --data '
    {
      "name": "ai-rate-limiting-advanced",
      "config": {
        "llm_providers": [
          {
            "name": "openai",
            "limit": [
              100,
              1000
            ],
            "window_size": [
              60,
              3600
            ]
          }
        ]
      }
    }
    '

Make sure to replace the following placeholders with your own values:

consumerName|Id: The id or name of the consumer the plugin configuration will target.

Make the following request:

curl -X POST https://{region}.api.konghq.com/v2/control-planes/{controlPlaneId}/core-entities/consumers/{consumerId}/plugins/ \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "name": "ai-rate-limiting-advanced",
      "config": {
        "llm_providers": [
          {
            "name": "openai",
            "limit": [
              100,
              1000
            ],
            "window_size": [
              60,
              3600
            ]
          }
        ]
      }
    }
    '

Make sure to replace the following placeholders with your own values:

region: Geographic region where your Kong Konnect is hosted and operates.
controlPlaneId: The id of the control plane.
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account.
consumerId: The id of the consumer the plugin configuration will target.

See the Konnect API reference to learn about region-specific URLs and personal access tokens.

echo "
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: ai-rate-limiting-advanced
  namespace: kong
  annotations:
    kubernetes.io/ingress.class: kong
config:
  llm_providers:
  - name: openai
    limit:
    - 100
    - 1000
    window_size:
    - 60
    - 3600
plugin: ai-rate-limiting-advanced
" | kubectl apply -f -

Next, apply the KongPlugin resource by annotating the KongConsumer resource:

kubectl annotate -n kong  CONSUMER_NAME konghq.com/plugins=ai-rate-limiting-advanced

Prerequisite: Configure your Personal Access Token

terraform {
  required_providers {
    konnect = {
      source  = "kong/konnect"
    }
  }
}

provider "konnect" {
  personal_access_token = "$KONNECT_TOKEN"
  server_url            = "https://us.api.konghq.com/"
}

Add the following to your Terraform configuration to create a Konnect Gateway Plugin:

resource "konnect_gateway_plugin_ai_rate_limiting_advanced" "my_ai_rate_limiting_advanced" {
  enabled = true

  config = {
    llm_providers = [
      {
        name = "openai"
        limit = [100, 1000]
        window_size = [60, 3600]
      }    ]
  }

  control_plane_id = konnect_gateway_control_plane.my_konnect_cp.id
  consumer = {
    id = konnect_gateway_consumer.my_consumer.id
  }
}

Add this section to your declarative configuration file:

_format_version: "3.0"
plugins:
  - name: ai-rate-limiting-advanced
    consumer_group: consumerGroupName|Id
    config:
      llm_providers:
      - name: openai
        limit:
        - 100
        - 1000
        window_size:
        - 60
        - 3600

Make sure to replace the following placeholders with your own values:

consumerGroupName|Id: The id or name of the consumer group the plugin configuration will target.

Make the following request:

curl -i -X POST http://localhost:8001/consumer_groups/{consumerGroupName|Id}/plugins/ \
    --header "Accept: application/json" \
    --header "Content-Type: application/json" \
    --data '
    {
      "name": "ai-rate-limiting-advanced",
      "config": {
        "llm_providers": [
          {
            "name": "openai",
            "limit": [
              100,
              1000
            ],
            "window_size": [
              60,
              3600
            ]
          }
        ]
      }
    }
    '

Make sure to replace the following placeholders with your own values:

consumerGroupName|Id: The id or name of the consumer group the plugin configuration will target.

Make the following request:

curl -X POST https://{region}.api.konghq.com/v2/control-planes/{controlPlaneId}/core-entities/consumer_groups/{consumerGroupId}/plugins/ \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "name": "ai-rate-limiting-advanced",
      "config": {
        "llm_providers": [
          {
            "name": "openai",
            "limit": [
              100,
              1000
            ],
            "window_size": [
              60,
              3600
            ]
          }
        ]
      }
    }
    '

Make sure to replace the following placeholders with your own values:

region: Geographic region where your Kong Konnect is hosted and operates.
controlPlaneId: The id of the control plane.
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account.
consumerGroupId: The id of the consumer group the plugin configuration will target.

See the Konnect API reference to learn about region-specific URLs and personal access tokens.

echo "
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: ai-rate-limiting-advanced
  namespace: kong
  annotations:
    kubernetes.io/ingress.class: kong
config:
  llm_providers:
  - name: openai
    limit:
    - 100
    - 1000
    window_size:
    - 60
    - 3600
plugin: ai-rate-limiting-advanced
" | kubectl apply -f -

Next, apply the KongPlugin resource by annotating the KongConsumerGroup resource:

kubectl annotate -n kong  CONSUMERGROUP_NAME konghq.com/plugins=ai-rate-limiting-advanced

Prerequisite: Configure your Personal Access Token

terraform {
  required_providers {
    konnect = {
      source  = "kong/konnect"
    }
  }
}

provider "konnect" {
  personal_access_token = "$KONNECT_TOKEN"
  server_url            = "https://us.api.konghq.com/"
}

Add the following to your Terraform configuration to create a Konnect Gateway Plugin:

resource "konnect_gateway_plugin_ai_rate_limiting_advanced" "my_ai_rate_limiting_advanced" {
  enabled = true

  config = {
    llm_providers = [
      {
        name = "openai"
        limit = [100, 1000]
        window_size = [60, 3600]
      }    ]
  }

  control_plane_id = konnect_gateway_control_plane.my_konnect_cp.id
  consumer_group = {
    id = konnect_gateway_consumer_group.my_consumer_group.id
  }
}

AI Rate Limiting Advanced

Enable LLM provider rate limitingv3.7+

Prerequisites

Set up the plugin

Help us make these docs great!

Still need help