Virtual clusters

What is a virtual cluster?

Virtual clusters are the primary way clients interact with the Event Gateway proxy. They allow you to isolate clients from each other when connecting to the same backend cluster, and provide each client with modified view while still appearing as a standard Kafka cluster.

The virtual cluster workflow operates as follows:

  1. The Kafka client produces a request.
  2. A listener forwards it to the correct virtual cluster.
  3. The virtual cluster applies policies and proxies the modified request to the backend cluster.
  4. The backend cluster, representing a Kafka cluster, receives the request and sends a response.
 
flowchart LR
    
    A[Kafka 
    client]
    B[Listener 
    &#40TCP socket&#41
    + listener policies]
    C@{ shape: processes, label: "Virtual clusters
    + consume, produce, 
    and cluster policies"}
    D[Backend 
    cluster]
    E[Kafka 
    cluster]

    A --> B
    subgraph id1 [Event Gateway]
    B --> C 
    C --> D
    end
    D --> E

    style C stroke:#86e2cc

    style id1 rx:7,ry:7
  

Note: Each virtual cluster can only expose one backend cluster, but you can have multiple virtual clusters connected to one backend. In other words, a single virtual cluster can’t aggregate data from multiple backend clusters.

Why use a virtual cluster?

Virtual clusters let you apply governance and security features to event streams. This way, a single Kafka cluster can be sliced into multiple access points, each with its own security policy.

Use case

Description

Policy enforcement Define policies on virtual clusters to govern client behavior. Policies include transformations, filtering, enforcing encryption and decryption standards, access control, and more.
Authentication and mediation Use authentication mediation to control access between clients and backend clusters. Event Gateway authenticates clients (for example, with OAuth tokens) and re-authenticates separately when forwarding requests to the backend.
Topic and cluster virtualization Use topic and cluster virtualization to simplify change management and security. Virtual clusters can expose only a subset of topics on the backend cluster.
Namespacing and topic rewriting Virtual clusters support namespaces, which rewrite and enforce consistent prefixes for topic and consumer group names. This allows you to expose clean, simple names to clients while maintaining organization on the backend.
Infrastructure planning Through logical isolation, virtual clusters help organizations reduce Kafka infrastructure costs, as they eliminate the need to maintain multiple physical Kafka clusters for environment or team separation.

Managing multiple environments or products

You will need to increase the number of virtual clusters if you want to create multiple environments or products on top of the same physical cluster.

Here are some common patterns:

  • Environment isolation: You can create isolated dev, test, and prod namespaces on top of the same physical Kafka cluster. If you have a topic named orders in each virtual cluster, this will map transparently to different backend topics: dev-orders, test-orders, and prod-orders. This provides isolation and automatic name resolution per environment.

    When clients create new topics through a virtual cluster using Kafka’s CreateTopics request, the namespace prefix is automatically applied, ensuring that clients always stay within their designated namespace.

  • External partner isolation: You can expose the same backend topic to different external partners with data filtering. For instance, a single orders topic can be exposed through separate virtual clusters (customer-a, customer-b, customer-c), with each customer seeing only their own orders.

  • Reverse mapping: One backend topic (orders) can appear as multiple separate topics (dev-orders, test-orders, prod-orders) across different virtual clusters, each pre-filtered for specific users.

Authentication

Authentication on the virtual cluster is used to authenticate clients to the proxy. The virtual cluster supports multiple authentication methods and can mediate authentication between clients and backend clusters. See Backend cluster authentication to learn more.

Virtual clusters support the following auth methods:

Auth method (authentication.type)

Description

Credential mediation types (authentication.mediation)

Anonymous (anonymous) Doesn’t require clients to provide any authentication when connecting to the proxy. None
SASL/PLAIN (sasl_plain) Requires clients to provide a username and password.

Accepts a hardcoded list of usernames and passwords, either as strings or environment variables.
passthrough, terminate
SASL/OAUTHBEARER (oauth_bearer) Requires clients to provide an OAuth token and a JWKS endpoint to verify token signatures, optionally with claim mapping and validation rules. passthrough, terminate, validate_forward
SASL/SCRAM-SHA-256 (sasl_scram) Requires clients to provide a username and password using SCRAM-SHA-256 hashing. Set authentication.algorithm to sha256. passthrough
SASL/SCRAM-SHA-512 (sasl_scram) Requires clients to provide a username and password using SCRAM-SHA-512 hashing. Set authentication.algorithm to sha512. passthrough
mTLS (client_certificate) v1.1+ Requires clients to present a trusted TLS certificate during the TLS handshake. The client certificate is verified against a TLS trust bundle. The certificate’s principal (for example, the Common Name) can be used to enforce access control with ACL policies. None

Credential mediation

With virtual clusters, you can control how client credentials are handled between the proxy and backend cluster, and reuse existing credentials and principals defined on the backend cluster.

Use the virtual cluster authentication.mediation setting to configure a mediation mode. Choose the mode based on your security requirements and backend cluster configuration:

  • Passthrough (passthrough): Authentication from the client passes through the proxy to the backend without validation. This method is required for SCRAM authentication.
  • Terminate (terminate): Checks whether the client’s connection is authorized based on their credential, and then terminates the authentication. Then, a new authentication session starts with the backend cluster.
  • Validate and forward (validate_forward): The client’s OAuth token is first validated by the proxy, and then sent to the backend as-is. This will “fail fast” if the token is invalid before sending it to the backend.

Namespaces

With namespaces, you can preserve any naming systems that you have in place, and ensure they remain consistent. Namespaces let you:

  • Rewrite and enforce topics, consumer groups, and transaction IDs with a consistent prefix
  • Expose topics and consumer groups through the virtual cluster

This allows you to expose clean, simple names to clients while maintaining organization on the backend.

For example, a virtual cluster exposes a topic named orders to the client. Behind the scenes, this maps to team-a-orders on the actual Kafka cluster. The client doesn’t need to know about or manage the team-a- prefix. This enables transparent multitenancy, where multiple teams can share the same Kafka cluster without needing to manually prefix every topic and consumer group name in their applications.

The following examples provide some common use cases for namespaces and show how to set them up.

Apply prefixes automatically

The most common use case for namespaces is to automatically prefix read and create operations when interacting with topics and consumer groups. This helps avoid overlapping from multiple tenants.

You can do this by setting a prefix on the virtual cluster:

curl -X POST "https://us.api.konghq.com/v1/event-gateways/$EVENT_GATEWAY_ID/virtual-clusters" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "name": "example-virtual-cluster",
       "destination": {
         "name": "example-backend-cluster"
       },
       "authentication": [
         {
           "type": "anonymous"
         }
       ],
       "dns_label": "virtual-cluster-1",
       "acl_mode": "passthrough",
       "namespace": {
         "mode": "hide_prefix",
         "prefix": "my-prefix"
       }
     }'

In this example, the prefix my-prefix will be used for all consumer group and topics that connect to this virtual cluster.

Access additional topics

Along with topics owned by a specific team, you can pull in a select group of additional topics. This is useful when you want to:

  • Consume topics owned by other teams
  • Gradually migrate to a namespace while still using old topics temporarily

Here’s an example configuration using a glob pattern:

curl -X POST "https://us.api.konghq.com/v1/event-gateways/$EVENT_GATEWAY_ID/virtual-clusters" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "name": "example-virtual-cluster",
       "destination": {
         "name": "example-backend-cluster"
       },
       "authentication": [
         {
           "type": "anonymous"
         }
       ],
       "dns_label": "virtual-cluster-1",
       "acl_mode": "passthrough",
       "namespace": {
         "mode": "hide_prefix",
         "prefix": "my-prefix",
         "additional": {
           "topics": [
             {
               "type": "glob",
               "glob": "my-topic-*",
               "conflict": "warn"
             }
           ]
         }
       }
     }'

These topics are accessed using their full unmodified names.

This example uses a glob expression to capture topics using name patterns. You can also pass an exact list of topics as an array:

"topics": [
  {
    "type": "exact_list",
    "list": [
      {
        "backend": "allowed_topic",
        "backend": "another_allowed_topic"
      }
    ]
  }
]

Access additional consumer groups

You can access existing consumer groups to avoid migrating offsets:

curl -X POST "https://us.api.konghq.com/v1/event-gateways/$EVENT_GATEWAY_ID/virtual-clusters" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "name": "example-virtual-cluster",
       "destination": {
         "name": "example-backend-cluster"
       },
       "authentication": [
         {
           "type": "anonymous"
         }
       ],
       "dns_label": "virtual-cluster-1",
       "acl_mode": "passthrough",
       "namespace": {
         "mode": "hide_prefix",
         "prefix": "my-prefix",
         "additional": {
           "consumer_groups": [
             {
               "type": "glob",
               "glob": "my-app-*",
               "conflict": "warn"
             }
           ]
         }
       }
     }'

End users of this virtual cluster can use their existing, unnamespaced consumer groups.

This example uses a glob expression to capture consumer groups using name patterns. You can also pass an exact list of consumer groups as an array:

"consumer_groups": [
  {
    "type": "exact_list",
    "list": [
      {
        "value": "foo",
        "value": "bar"
      }
    ]
  }
]

Topic aliases v1.2+

Topic aliases let you expose a backend Kafka topic under a different, client-facing name. Clients connecting to the virtual cluster see the alias, while Event Gateway transparently routes requests to the original backend topic.

This is useful when you want to:

  • Expose internally-named topics (like team-alpha-orders-v2) under client-friendly names (like orders)
  • Migrate clients to renamed backend topics without updating client configuration
  • Provide multiple client-facing names for the same backend topic

Unlike namespaces, which apply a prefix to every topic and consumer group, topic aliases are explicit one-to-one mappings between an alias and a backend topic. Both the alias and the original backend topic name remain accessible through the virtual cluster.

Map a friendly name to a backend topic

Define topic aliases on the virtual cluster using the topic_aliases field:

curl -X POST "https://us.api.konghq.com/v1/event-gateways/$EVENT_GATEWAY_ID/virtual-clusters" \
     --no-progress-meter --fail-with-body  \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "name": "example-virtual-cluster",
       "destination": {
         "name": "example-backend-cluster"
       },
       "authentication": [
         {
           "type": "anonymous"
         }
       ],
       "dns_label": "virtual-cluster-1",
       "acl_mode": "passthrough",
       "topic_aliases": [
         {
           "alias": "orders",
           "topic": "team-alpha-orders-v2"
         },
         {
           "alias": "clicks",
           "topic": "analytics-raw-clicks"
         }
       ]
     }'

In this example, clients can produce and consume to orders and clicks. Event Gateway resolves these to the backend topics team-alpha-orders-v2 and analytics-raw-clicks.

Supported operations

Aliases are a read-only abstraction over physical topics. Read and write operations are forwarded to the backend topic, but topic-modifying operations are rejected.

Operation

Behavior

Produce, fetch, list offsets, consumer group operations Allowed. Requests reference the alias name and are transparently resolved to the backend topic.
Metadata (ListTopics) Allowed. Both the alias and the original backend topic name appear in the response.
CreateTopics, DeleteTopics, CreatePartitions, DeleteRecords, AlterPartitionReassignments, ElectLeaders Rejected with InvalidTopicException when the request references an alias. This avoids a client unintentionally modifying a physical topic that other aliases or clients depend on.

ACLs are evaluated on the name the client uses, before alias resolution. An ACL on the backend topic does not automatically grant access to its aliases, and vice versa. Operators must configure ACLs for each alias name explicitly. With acl_mode: enforce_on_gateway (deny-by-default), a new alias with no matching ACL is blocked.

Interaction with namespaces

The topic field in topic_aliases references namespace-visible names, not raw backend topic names. In the request path, aliases resolve after policies but before the namespace transformer:

client → policies → alias resolution → namespace → backend

If the virtual cluster has a namespace, the alias must target a name that the namespace already exposes. For example, with namespace.mode: hide_prefix and prefix: analytics_, topic: foo resolves to the backend topic analytics_foo.

A backend topic that the namespace doesn’t expose (not matching the prefix and not listed in additional_topics) can’t be aliased directly. Expose it through the namespace first:

namespace:
  mode: hide_prefix
  prefix: analytics_
  additional_topics:
    - type: exact_list
      list:
        - team-alpha-orders-v2
topic_aliases:
  - alias: orders
    topic: team-alpha-orders-v2 # valid because additional_topics exposes it

Policy context

Policies attached to a virtual cluster see the alias name, not the backend topic name. In a CEL match expression, topic.name evaluates to the alias name. Policies that match on a backend topic name don’t automatically apply to its aliases. Match the alias name explicitly:

condition: topic.name == "orders"

Consumer groups

Avoid consuming from both an alias and its backend topic with the same consumer group. Event Gateway treats the alias and the backend topic as distinct topic names that share the same underlying partitions. Mixing them in one consumer group leaves offset commits and partition assignment undefined. Pick one name per consumer group and stick with it.

For a full walkthrough, see Configure topic aliases.

Virtual cluster policies

Virtual clusters can be modified by policies, which let you do things like modify headers, encrypt and decrypt records, validate record schemas, and much more.

To learn more, see:

Set up a virtual cluster

Before setting up a virtual cluster, make sure you have a backend cluster configured. A virtual cluster must connect to an existing backend cluster.

Schema

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!