Confluent Consume Policy

Premium Partner

Clients that need Kafka event data, such as audit events or notifications, usually need a Kafka client library. The Confluent Consume AI Policy reads messages from Confluent Cloud Kafka topics and serves them from an AI Gateway endpoint over HTTP, server-sent events, or WebSocket. Attach it to an AI Model, AI Agent, or AI MCP Server to expose the topics on that entity’s route.

AI Gateway also provides the Confluent Policy for publishing messages to Confluent Cloud.

Consume messages

The Policy serves messages on the route of the entity it’s attached to. For an AI Model, send a POST request to the model’s chat/completions path. For an AI Agent or AI MCP Server, send a GET request to the entity’s path.

In http-get mode, the Policy returns the records available for every topic in config.topics, keyed by topic name and then by partition:

{
  "ai-events": {
    "partitions": {
      "0": {
        "high_watermark": 3,
        "last_stable_offset": 3,
        "errcode": 0,
        "records": [
          {
            "value": {"event": "prompt_received", "n": 1},
            "offset": 0,
            "timestamp": 1794215129672
          }
        ],
        "aborted_transactions": {}
      }
    }
  }
}

When config.message_deserializer is json, value is a parsed object. When it’s noop (default), value is the raw message as a string.

Offsets and repeated reads

The Confluent Consume Policy reads topics as a Kafka consumer group. By default, the Policy commits offsets after each read, so a repeated request returns only new messages. When no new messages exist, records is an empty object.

Where a new consumer group starts reading depends on config.auto_offset_reset:

  • earliest (default) reads from the beginning of the topic.
  • latest reads only messages produced after the Policy starts consuming. Also set config.enforce_latest_offset_reset to true. Without it, the first read still returns the messages already on the topic.

Modes

Set config.mode to choose how the Policy delivers messages:

  • http-get (default): Returns the available messages in one response.
  • server-sent-events: Streams each message as a server-sent event with the fields data, event, id, and topic, and sends keep-alive comments while no messages arrive. Clients typically send Accept: text/event-stream.
  • websocket: Sends each message as a JSON text frame over a WebSocket connection. Each frame has the fields data, event, id, and topic.

Transform messages

Use config.message_by_lua_functions to run custom Lua code on each message in server-sent-events mode. The function receives each message as a table with fields such as value and returns the modified message. String and boolean fields you add to the returned message appear as extra fields on the event.

The following function marks every message as processed:

return function(message)
  local m = type(message) == "table" and message or { value = message }
  m.processed_by_lua = true
  return m
end

Limitations

Messages that fail JSON deserialization aren’t sent to the dead-letter topic, even when config.enable_dlq and config.dlq_topic are set.

Scopes

Attach the Confluent Consume Policy to an AI Model, AI Agent, AI MCP Server, or AI Consumer to serve Kafka messages for that entity only, or apply it globally. For an AI Consumer, the Policy takes effect only after the client authenticates.

For details about how scopes work, see AI Policy scopes.

Looking for on-prem configuration? Convert the kongctl configuration to a decK yaml file and use deck file ai2kong. See Configure Kong AI Gateway on-prem for more information.

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!