Kafka ACLs are surprisingly effective at controlling access at a granular level, but they often get misunderstood as a simple allow/deny switch.
Let’s see Kafka ACLs in action. Imagine we have a Kafka cluster and we want to restrict access to a topic named sensitive-data. Only a specific application, let’s call it data-processor, should be allowed to read from it, and another application, data-writer, should be allowed to write to it.
First, we need to identify the principal (user) for each application. In Kafka, principals are typically represented as strings, often in the format User:username. Let’s assume data-processor runs as User:processor-user and data-writer runs as User:writer-user.
To allow data-writer to produce messages to sensitive-data, we’d use a command like this:
kafka-acls.sh --bootstrap-server kafka-broker-1:9092 --add --allow-principal User:writer-user --operation Write --topic sensitive-data
This command tells Kafka: "For the topic sensitive-data, grant User:writer-user the Write permission."
Now, let’s allow data-processor to consume messages from sensitive-data:
kafka-acls.sh --bootstrap-server kafka-broker-1:9092 --add --allow-principal User:processor-user --operation Read --topic sensitive-data
This command grants User:processor-user the Read permission on sensitive-data.
But what about consumer groups? If data-processor needs to consume from sensitive-data using a consumer group named processor-group, we need to grant it permission to Read from that group. This is crucial because consumer group management (like committing offsets) is also an operation controlled by ACLs.
kafka-acls.sh --bootstrap-server kafka-broker-1:9092 --add --allow-principal User:processor-user --operation Read --group processor-group
This grants User:processor-user the ability to read metadata and commit offsets for the processor-group consumer group.
The mental model for Kafka ACLs revolves around a few key concepts:
- Principals: Who is requesting access? This is typically a user (e.g.,
User:kafka-user,User:admin) or a service principal in more complex security setups. - Resources: What are they trying to access? This can be a topic, a consumer group, a cluster-wide operation (like
Describe), or even transactional IDs. - Operations: What are they trying to do? Common operations include
Read,Write,Create,Delete,Describe,DescribeConfigs,Alter,ClusterAction. - Permissions: The combination of a Principal, Resource, Operation, and a
AlloworDenyrule. Kafka evaluates these rules to determine if access is granted.
When a client connects to Kafka and attempts an operation, Kafka checks its ACLs. The rules are evaluated in a specific order. If a client is denied access by an explicit Deny rule, access is immediately revoked. If no Deny rule matches, Kafka then looks for an Allow rule. If an Allow rule matches, access is granted. If neither a Deny nor an Allow rule matches, the default behavior is to deny access, unless the allow.everyone.if.no.acl.found broker configuration is set to true.
The kafka-acls.sh tool is your primary interface for managing these. You can also use --list to see existing ACLs and --remove to delete them. It’s important to remember that ACLs are applied at the broker level, and Kafka relies on ZooKeeper (or its internal KRaft quorum) to store and distribute them.
A common point of confusion is the difference between topic-level Read and group-level Read. Topic Read allows a consumer to fetch messages from partitions. Group Read allows a consumer within a specific consumer group to perform operations related to that group, such as committing offsets. Without group Read permission, a consumer might be able to fetch messages but won’t be able to track its progress correctly, leading to re-processing of messages or inability to start consuming from a specific offset.
Once you’ve got your ACLs set up, you’ll likely run into authorization issues related to cluster-wide operations or specific configurations.