Confluent Consume

This plugin consumes messages from Confluent Cloud Kafka topics and makes them available through HTTP endpoints. For more information, see the Confluent Cloud documentation.

Note: This plugin has the following known limitations:

• Message compression is not supported.
• The message format is not customizable.
• Kong Gateway does not support Kafka 4.0.

Kong also provides a plugin for publishing messages to Confluent Cloud.

The plugin supports the following modes of operation:

• http-get: Consume messages via an HTTP GET requests (default)
• server-sent-events: Stream messages using server-sent events

When running multiple Data Plane nodes, there is no thread-safe behavior between nodes. In high-load scenarios, you may observe the same message being delivered multiple times across different Data Plane nodes

To minimize duplicate message delivery in a multi-node setup, consider:

• Using a single Data Plane node for consuming messages from specific topics
• Implementing idempotency handling in your consuming application
• Monitoring Consumer Group offsets across your Data Plane nodes

The Confluent Consume plugin supports integration with Confluent Schema Registry for AVRO and JSON schemas.

Schema registries provide a centralized repository for managing and validating schemas for data formats like AVRO and JSON. Integrating with a schema registry allows the plugin to validate and serialize/deserialize messages in a standardized format.

Using a schema registry with Kong Gateway provides several benefits:

• Data validation: Ensures messages conform to a predefined schema before being processed.
• Schema evolution: Manages schema changes and versioning.
• Interoperability: Enables seamless communication between different services using standardized data formats.
• Reduced overhead: Minimizes the need for custom validation logic in your applications.

To learn more about Kong’s supported schema registry, see:

• Confluent Schema Registry Documentation
• AVRO Specification

How schema registry validation works

When a consumer plugin is configured with a schema registry, the following workflow occurs:

 
sequenceDiagram
autonumber
    participant Kafka
    participant Kong as Confluent Consume plugin
    participant Registry as Schema Registry
    participant Client
    
    activate Kafka
    activate Kong
    Kafka->>Kong: Send message
    deactivate Kafka
    Kong->>Kong: Extract schema ID
    activate Registry
    Kong->>Registry: Fetch schema from registry
    Registry-->>Kong: Return schema
    deactivate Registry
    Kong->>Kong: Deserialize using schema
    activate Client
    Kong->>Client: Return response to client
    deactivate Kong
    deactivate Client
  

Configure schema registry

To configure Schema Registry with the Confluent Consume plugin, use the config.schema_registry parameter in your plugin configuration.

See the schema registry configuration example for sample configuration values.