Llama provider

You can proxy requests to Llama2 AI models through AI Gateway by creating AI Model Provider and AI Model entities. This reference documents all supported AI capabilities, configuration requirements, and provider-specific details needed for proper integration.

AI Gateway automatically routes requests to the appropriate Llama2 API endpoints. The following table shows the upstream paths used for each capability.

The following tables show the AI capabilities supported by the Llama2 provider when configuring AI Models.

By default, AI Gateway uses the path templates shown in the tables below (e.g., /chat/completions, /embeddings, etc.). To customize these paths, configure the config.paths field in your AI Model entity. Custom paths take the form {configured_path}/{template_path} — for example, if you set a custom path of /v2, requests to /embeddings would be routed to /v2/embeddings.

Text generation

Support for Llama2 text generation capabilities:

Embeddings

Support for Llama2 embeddings generation:

The base URL is $UPSTREAM_URL.

AI Gateway uses this URL automatically. You only need to configure a URL if you’re using a self-hosted or Llama2-compatible endpoint, in which case set the upstream_url option in your AI Model configuration.

To use Llama2 with AI Gateway, configure a new AI Model Provider. You can then access supported AI Models from Llama2.

Here’s a minimal configuration for chat completions:

curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/model-providers \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "display_name": "Llama2 Production",
      "name": "my-llama2-account",
      "type": "llama2",
      "config": {
        "auth": {
          "type": "basic",
          "headers": [
            {
              "name": "Authorization",
              "value": "'$LLAMA_API_KEY'"
            }
          ]
        }
      }
    }
    '

ai_gateway_model_providers:
  - ref: my-llama2-account
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    display_name: Llama2 Production
    name: my-llama2-account
    type: llama2
    config:
      auth:
        type: basic
        headers:
        - name: Authorization
          value: !secret {source: !env LLAMA_API_KEY}

A target is an entry in the targets array on the AI Model entity, not the AI Model Provider. Beyond the common target options (name, provider, weight), a target routing to Llama2 requires:

• upstream_url: The URL of your self-hosted Llama model endpoint.
• format: The request format your endpoint expects. One of ollama, openai, or raw.

targets:
  - name: llama-3-70b
    provider: my-llama2-account
    config:
      type: llama2
      upstream_url: https://my-llama-endpoint.internal:8000
      format: openai