Kafka’s core innovation isn’t just about moving data; it’s about making data an immutable, append-only log that can be read and replayed by any number of consumers, at their own pace, without ever being deleted by the producer.
Let’s watch Kafka in action. Imagine we have a simple web application generating user clickstream data. We want to capture these events, process them for analytics, and maybe push them to a data warehouse.
First, we need a Kafka cluster. A minimal setup involves at least three brokers for fault tolerance.
# Example Kafka broker configuration (server.properties)
broker.id=0
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://kafka-broker-0.example.com:9092
log.dirs=/var/lib/kafka/data
zookeeper.connect=zookeeper-1.example.com:2181,zookeeper-2.example.com:2181,zookeeper-3.example.com:2181
# ... other configurations
We’ll create a topic named clickstream to hold our events. Topics are like categories or feeds for messages.
# Create a topic with 3 partitions and a replication factor of 3
kafka-topics.sh --create --topic clickstream --partitions 3 --replication-factor 3 --bootstrap-server kafka-broker-0.example.com:9092
Now, a producer application can start sending messages to this clickstream topic. Each message is a key-value pair, where the key is optional and the value is the actual data.
// Example Producer Code (Java)
import org.apache.kafka.clients.producer.*;
import java.util.Properties;
public class ClickstreamProducer {
public static void main(String[] args) {
Properties props = new Properties();
props.put("bootstrap.servers", "kafka-broker-0.example.com:9092,kafka-broker-1.example.com:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
Producer<String, String> producer = new KafkaProducer<>(props);
for (int i = 0; i < 100; i++) {
String userId = "user_" + (i % 10);
String event = "{\"timestamp\": " + System.currentTimeMillis() + ", \"userId\": \"" + userId + "\", \"action\": \"click\"}";
// Using userId as the key to ensure all events for a user go to the same partition
producer.send(new ProducerRecord<>("clickstream", userId, event), (metadata, exception) -> {
if (exception == null) {
System.out.println("Sent message to topic " + metadata.topic() + " partition " + metadata.partition() + " offset " + metadata.offset());
} else {
System.err.println("Error sending message: " + exception.getMessage());
}
});
}
producer.close();
}
}
When the producer sends a message with a userId key, Kafka’s partitioning mechanism ensures all messages with the same key land in the same partition. This is crucial for ordered processing within a partition. For our clickstream topic with 3 partitions, Kafka will hash the userId and map it to one of the partitions (0, 1, or 2).
Meanwhile, multiple consumer applications can read from this topic. A consumer group allows multiple instances of an application to work together, distributing the partitions among themselves.
// Example Consumer Code (Java)
import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;
public class ClickstreamConsumer {
public static void main(String[] args) {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-broker-0.example.com:9092,kafka-broker-1.example.com:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "clickstream-analytics"); // Consumer group ID
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); // Start from the beginning if no offset is found
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Collections.singletonList("clickstream")); // Subscribe to the topic
System.out.println("Waiting for messages...");
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
for (java.util.Map.Entry<org.apache.kafka.common.TopicPartition, ConsumerRecords<String, String>> entry : records.entrySet()) {
for (ConsumerRecord<String, String> record : entry.getValue()) {
System.out.printf("Received message: topic = %s, partition = %d, offset = %d, key = %s, value = %s%n",
record.topic(), record.partition(), record.offset(), record.key(), record.value());
// Here you would process the clickstream event for analytics
}
}
// Committing offsets is crucial. In a real application, you'd do this periodically
// or after processing a batch of messages.
// consumer.commitSync();
}
// consumer.close(); // This loop is infinite, so close is never reached in this example
}
}
The ClickstreamConsumer belongs to the clickstream-analytics consumer group. If we run multiple instances of this consumer with the same GROUP_ID_CONFIG, Kafka automatically distributes the 3 partitions of the clickstream topic among them. If one consumer instance goes down, Kafka rebalances the partitions to the remaining instances. The offset is a pointer to the last message read by a consumer within a partition. Kafka stores these offsets, allowing consumers to resume from where they left off.
Kafka achieves its distributed nature through a combination of brokers, topics, partitions, and replication. Each topic is divided into partitions, which are ordered, immutable sequences of records. Partitions are the unit of parallelism in Kafka. A topic with more partitions can support higher throughput. Replication ensures fault tolerance: each partition has a leader broker and one or more follower brokers. If the leader fails, one of the followers is elected as the new leader.
The "magic" of Kafka lies in its ability to decouple producers and consumers. Producers write to topics, and consumers read from topics. Consumers manage their own progress (offsets) and can read data at their own speed, re-read old data (as long as it hasn’t been purged based on retention policies), or even be added to an existing consumer group to catch up on processing. This makes it incredibly flexible for building real-time data pipelines, microservice communication, and event sourcing systems.
The most surprising thing about Kafka’s durability is that it’s not just about keeping data around; it’s about the order within partitions being guaranteed, but not necessarily across partitions unless you use a key. If you send messages with the same key, they will always land in the same partition and thus be processed in the order they were sent. This is how you can achieve strict ordering for specific entities, like ensuring all actions for a single user are processed sequentially.
The next step is understanding how to manage data retention and explore more advanced consumer coordination mechanisms like manual commits and idempotent producers.