Get started with AI Gateway

Uses: AI Gateway Kong Gateway decK

Deployment Platform:

Prerequisites

Kong Konnect

This is a Konnect tutorial. If you don’t have a Konnect account, you can get started quickly with our onboarding wizard.

The following Konnect items are required to complete this tutorial:
- Personal access token (PAT): Create a new personal access token by opening the Konnect PAT page and selecting Generate Token.
- Control Plane Name: You can use an existing Control Plane or create a new one to use for this tutorial.
- Konnect Proxy URL: By default, a self-hosted Data Plane uses http://localhost:8000. You can set up Data Plane nodes for your Control Plane from the Gateway Manager in Konnect.

Set the personal access token, the Control Plane name, the Control Plane URL, and the Konnect proxy URL as environment variables:

 export DECK_KONNECT_TOKEN='YOUR KONNECT TOKEN'
 export DECK_KONNECT_CONTROL_PLANE_NAME='YOUR CONTROL PLANE NAME'
 export KONNECT_CONTROL_PLANE_URL=https://us.api.konghq.com
 export KONNECT_PROXY_URL='KONNECT PROXY URL'

Kong Gateway running

This tutorial requires Kong Gateway Enterprise. If you don’t have Kong Gateway set up yet, you can use the quickstart script with an enterprise license to get an instance of Kong Gateway running almost instantly.

Export your license to an environment variable:

 export KONG_LICENSE_DATA='LICENSE-CONTENTS-GO-HERE'

Run the quickstart script:

 curl -Ls https://get.konghq.com/quickstart | bash -s -- -e KONG_LICENSE_DATA 

Once Kong Gateway is ready, you will see the following message:

 Kong Gateway Ready

decK

decK is a CLI tool for managing Kong Gateway declaratively with state files. To complete this tutorial you will first need to install decK.

OpenAI

This tutorial uses the AI Proxy plugin with OpenAI. You’ll need to create an OpenAI account and get an API key. Once you have your API key, create an environment variable:

export OPENAI_KEY='<api-key>'

Check that Kong Gateway is running

We’ll be using decK for this tutorial, so let’s check that Kong Gateway is running and that decK can access it:

deck gateway ping

If everything is running, then you should get the following response:

Successfully connected to Kong!
Kong version: 3.9.0.0

Successfully Konnected to the Kong organization!

Create a Gateway Service

Create a Service to contain the Route for the LLM provider:

echo '
_format_version: "3.0"
services:
  - name: llm-service
    url: http://localhost:32000
' | deck gateway apply -

The URL can point to any empty host, as it won’t be used by the plugin.

Create a Route

Create a Route for the LLM provider. In this example we’re creating a chat route, so we’ll use /chat as the path:

echo '
_format_version: "3.0"
routes:
  - name: openai-chat
    service:
      name: llm-service
    paths:
    - "/chat"
' | deck gateway apply -

Enable the AI Proxy plugin

Enable the AI Proxy plugin to create a chat route:

echo '
_format_version: "3.0"
plugins:
  - name: ai-proxy
    config:
      route_type: llm/v1/chat
      model:
        provider: openai
' | deck gateway apply -

In this example, we’re setting up the plugin with minimal configuration, which means:

The client is allowed to use any model in the openai provider and must provide the model name in the request body.
The client must provide an Authorization header with an OpenAI API key.

If needed, you can restrict the models that can be consumed by specifying the model name explicitly using the config.model.name parameter.

You can also provide the OpenAI API key directly in the configuration with the config.auth.header_name and config.auth.header_value parameters so that the client doesn’t have to send them.

Validate

To validate, you can send a POST request to the /chat endpoint, using the correct input format. Since we didn’t add the model name and API key in the plugin configuration, make sure to include them in the request:

curl -X POST "$KONNECT_PROXY_URL/chat" \
     -H "Accept: application/json"\
     -H "Content-Type: application/json"\
     -H "Authorization: Bearer $OPENAI_KEY" \
     --json '{
       "model": "gpt-4",
       "messages": [
         {
           "role": "user",
           "content": "Say this is a test!"
         }
       ]
     }'

curl -X POST "http://localhost:8000/chat" \
     -H "Accept: application/json"\
     -H "Content-Type: application/json"\
     -H "Authorization: Bearer $OPENAI_KEY" \
     --json '{
       "model": "gpt-4",
       "messages": [
         {
           "role": "user",
           "content": "Say this is a test!"
         }
       ]
     }'

You should get a 200 OK response, and the response body should contain This is a test.

Cleanup

Clean up Konnect environment

If you created a new control plane and want to conserve your free trial credits or avoid unnecessary charges, delete the new control plane used in this tutorial.

Destroy the Kong Gateway container

curl -Ls https://get.konghq.com/quickstart | bash -s -- -d

Next Steps

Set up load balancing using AI Proxy Advanced plugin

Cache traffic using the AI Semantic cache plugin

Secure traffic with the AI Prompt Guard

Provide prompt templates with AI Prompt Template

Programmatically inject system or assistant prompts to all incoming prompts with the AI Prompt Decorator

Learn about all the AI plugins