Kafka consumer groups allow multiple consumers to process messages from Kafka topics in parallel, effectively distributing the load and increasing throughput.
Let’s see this in action. Imagine a Kafka topic named user_events with 10 partitions. We have a producer sending events like user logins, page views, and purchases. Without consumer groups, a single consumer would have to process all these events sequentially. This can quickly become a bottleneck.
Here’s a simplified producer sending data:
kafka-console-producer --broker-list localhost:9092 --topic user_events
> {"user_id": 123, "event": "login", "timestamp": 1678886400}
> {"user_id": 456, "event": "page_view", "timestamp": 1678886405}
> {"user_id": 123, "event": "purchase", "timestamp": 1678886410}
Now, to process these events in parallel, we use consumer groups. A consumer group is a logical collection of consumers that work together to consume a topic. Each consumer within a group is assigned one or more partitions to process. Kafka guarantees that each partition is consumed by only one consumer within a specific group at any given time.
Consider a simple consumer configuration:
bootstrap.servers=localhost:9092
group.id=user_processing_group
key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
value.deserializer=org.apache.kafka.common.serialization.StringDeserializer
auto.offset.reset=earliest
If we start two instances of a consumer using this configuration, Kafka will automatically distribute the 10 partitions of user_events between them. For example, consumer A might get partitions 0-4, and consumer B might get partitions 5-9. Both consumers will then start processing messages from their assigned partitions.
The key benefit here is parallel processing. If each partition can be processed independently, and we have enough consumers in the group (up to the number of partitions), we can achieve a processing rate limited by the slowest consumer and the throughput of a single partition, rather than the total throughput of the topic.
The internal mechanism for this is partition assignment. When a consumer joins a group, it participates in a "rebalance" process. During a rebalance, the group coordinator (a broker designated to manage the group) re-allocates partitions among the active consumers. If a consumer leaves or a new one joins, another rebalance occurs. This dynamic assignment ensures that the workload is distributed.
The group.id is the crucial identifier. All consumers that share the same group.id are considered part of the same group and will coordinate their partition consumption. If you start a third consumer with the same group.id, it will trigger a rebalance, and partitions will be re-assigned again to distribute the load across three consumers.
The mental model to hold is that a topic produces messages, a consumer group consumes messages, and within that group, consumers (instances) are assigned partitions. A partition is the unit of parallelism. You can have as many consumers in a group as you want, but if the number of consumers exceeds the number of partitions, some consumers will be idle, waiting for a partition to become available (which only happens if another consumer in the group fails or leaves).
This partition-to-consumer mapping is managed by Kafka’s group coordination protocol. Consumers periodically send heartbeats to the group coordinator. If a consumer fails to send heartbeats for a configured period (session timeout), it’s considered dead, and its partitions are re-assigned to other active members of the group.
The auto.offset.reset setting determines where a consumer starts if it’s a new group or if its committed offset is lost. earliest means it starts from the very beginning of the topic’s history, while latest means it starts with messages produced after it joined.
The most surprising thing is how Kafka handles the "exactly-once" processing semantics with consumer groups, even though it’s fundamentally at-least-once delivery at the broker level. It achieves this through a combination of idempotent producers (if used), transactional producers, and consumer-side offset management. Consumers commit their offsets after successfully processing messages. If a consumer crashes mid-processing, it will re-read messages upon restart because its offset hasn’t been committed yet. The application logic then needs to handle potential duplicate processing, typically by making operations idempotent.
The next concept you’ll grapple with is how to manage the state of processed messages, especially when dealing with potential duplicates or ensuring transactional integrity across multiple Kafka topics or external systems.