Get started with AI Gateway

Create an AI Model Provider entity to define your connection to OpenAI and store your authentication credentials.

First, set the OPENAI_AUTH_HEADER environment variable to your OpenAI API key:

export OPENAI_AUTH_HEADER="Bearer $OPENAI_API_KEY"

Then, apply the configuration using kongctl:

kongctl apply -f - --auto-approve --pat "$KONNECT_TOKEN" << 'EOF'
ai_gateway_model_providers:
  - ref: generic-openai
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    name: generic-openai
    display_name: "generic-openai"
    type: openai
    config:
      auth:
        type: basic
        headers:
        - name: Authorization
          value: !secret {source: !env OPENAI_AUTH_HEADER}
EOF

!env AI_GATEWAY_ID references the AI Gateway created by the quickstart script in the prerequisites, instead of creating a new one.

In this example, we’re setting up the AI Model Provider with:

• type: openai: Specifies that this provider connects to the OpenAI service using OpenAI’s standard API format.
• name: generic-openai: A unique identifier that AI Models will reference to route requests through this provider.
• config.auth: Stores your OpenAI API key. AI Gateway securely manages this credential and injects it into upstream requests automatically, eliminating the need for clients to pass API keys.

Create an AI Model entity to declare which upstream models are available, configure how client requests are routed, and specify which AI Model Provider to use:

kongctl apply -f - --auto-approve --pat "$KONNECT_TOKEN" << 'EOF'
ai_gateway_models:
  - ref: my-gpt-4o
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    name: my-gpt-4o
    display_name: "my-gpt-4o"
    type: model
    formats:
      - type: openai
    config:
      route:
        paths:
          - /v1
        model:
          body_param: model
          values:
            - my-gpt-4o
    targets:
      - name: gpt-4o
        provider: generic-openai
        config:
          type: openai
    policies: []
    capabilities:
      - generate
EOF

!env AI_GATEWAY_ID references the AI Gateway created by the quickstart script, same as in the previous step.

In this example, we’re setting up the AI Model with:

• type: model: Specifies this is a synchronous model for request/response workloads.
• name: my-gpt-4o: A unique identifier for this model.
• formats: [type: openai]: Declares that this model accepts requests in OpenAI-compatible format.
• config.route.paths: [/v1]: Configures the custom base path for this model’s endpoints. Clients send requests to paths that combine this base path with capability-specific paths.
• capabilities: [generate]: Enables the text generation capability. The generate capability creates a /chat/completions endpoint, so combined with your base path, clients send chat requests to /v1/chat/completions.
• config.route.model: { body_param: model, values: [my-gpt-4o] }: Lets clients send my-gpt-4o in the request model field instead of the upstream model name.
• targets: Specifies which upstream AI Model Provider model to route requests to. Here, provider: generic-openai references the AI Model Provider we created earlier, and name: gpt-4o specifies which OpenAI model to call upstream.

Send a chat request to verify your setup:

curl -X POST "$KONNECT_PROXY_URL/v1/chat/completions" \
     --no-progress-meter --fail-with-body  \
     -H "Accept: application/json"\
     -H "Content-Type: application/json"\
     -H "Authorization: Bearer $OPENAI_API_KEY" \
     --json '{
       "messages": [
         {
           "role": "user",
           "content": "Say this is a test!"
         }
       ],
       "model": "my-gpt-4o"
     }'