Confluent

With Kafka at its core, Confluent offers complete, fully managed, cloud-native data streaming that’s available everywhere your data and applications reside. Using the Confluent plugin, you can send HTTP request data to Apache Kafka by constructing Kafka messages from incoming Kong Gateway HTTP requests.

Kong Gateway also provides Kafka Log and Kafka Upstream plugins for publishing logs and messages to an Apache Kafka topic:

• See Kafka Log
• See Kafka Upstream

Note: This plugin has the following known limitations:

• Message compression is not supported.
• The message format is not customizable.
• Kong Gateway supports Kafka 4.0 starting from version 3.10.

The Confluent plugin supports the following authentication options for Confluent Cloud connections:

The Confluent plugin supports integration with Confluent Schema Registry for AVRO and JSON schemas.

Schema registries provide a centralized repository for managing and validating schemas for data formats like AVRO and JSON. Integrating with a schema registry allows the plugin to validate and serialize/deserialize messages in a standardized format.

Using a schema registry with Kong Gateway provides several benefits:

• Data validation: Ensures messages conform to a predefined schema before being processed.
• Schema evolution: Manages schema changes and versioning.
• Interoperability: Enables seamless communication between different services using standardized data formats.
• Reduced overhead: Minimizes the need for custom validation logic in your applications.

To learn more about Kong’s supported schema registry, see:

• Confluent Schema Registry Documentation
• AVRO Specification

How schema registry validation works

When a producer plugin is configured with a schema registry, the following workflow occurs:

 
sequenceDiagram
autonumber
    participant Client
    participant Kong as Confluent plugin
    participant Registry as Schema Registry
    participant Kafka
    
    activate Client
    activate Kong
    Client->>Kong: Send request
    deactivate Client
    activate Registry
    Kong->>Registry: Fetch schema from registry
    Registry-->>Kong: Return schema
    deactivate Registry
    Kong->>Kong: Validate message against schema
    Kong->>Kong: Serialize using schema
    activate Kafka
    Kong->>Kafka: Forward to Kafka
    deactivate Kong
    deactivate Kafka
  

If validation fails, the request is rejected with an error message.

Configure schema registry

To configure Schema Registry with the Confluent plugin, use the config.schema_registry parameter in your plugin configuration.

For sample configuration values, see:

• Schema registry configuration example

The Confluent plugin can forward HTTP request headers as Kafka record headers, which are per-record key/value metadata that lives alongside the message key and value. This lets consumers read routing, tracing, or tenancy context without parsing the message payload.

Configure the config.headers block to control which headers are forwarded:

Use config.headers.name_mappings to rename an HTTP header to a different Kafka record header key.

Use config.headers.repeated_headers_behavior to control how duplicate HTTP headers are handled: retain_duplicates (default) creates a separate record header per value, take_first uses only the first value, and concatenate_by_comma joins all values with a comma.

Note: The config.forward_headers setting embeds request headers inside the message body. config.headers is a separate configuration block that sets native Kafka record headers on the produced record.