Kafka’s log retention policies are often misunderstood as a simple "delete old data" mechanism, but their true power lies in how they enable Kafka to act as a durable, replayable log for distributed systems.
Let’s see this in action. Imagine a Kafka topic named user_activity that’s getting hammered with events. We want to keep this data for a week, but also cap its total size to avoid filling up disks.
Here’s how a producer might write to this topic:
kafka-console-producer.sh --broker-list localhost:9092 --topic user_activity
> {"user_id": "alice", "event": "login", "timestamp": 1678886400}
> {"user_id": "bob", "event": "view_product", "timestamp": 1678886415}
And a consumer reading it:
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic user_activity --from-beginning
{"user_id": "alice", "event": "login", "timestamp": 1678886400}
{"user_id": "bob", "event": "view_product", "timestamp": 1678886415}
Now, let’s configure retention. This is done at the topic level, either during topic creation or by altering an existing topic.
Topic Creation with Retention Policies:
kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 3 --topic user_activity --config retention.ms=604800000 --config max.message.bytes=10485760
retention.ms=604800000: This sets the time-based retention to 7 days (604,800,000 milliseconds). Messages older than this will be eligible for deletion.max.message.bytes=10485760: This sets the maximum size of a single message to 10MB. While not directly a retention policy, it’s crucial for managing disk usage alongside time and size limits.
Altering Existing Topic Retention Policies:
If the user_activity topic already exists, you can change its retention like this:
kafka-topics.sh --alter --bootstrap-server localhost:9092 --topic user_activity --config retention.ms=259200000
This command changes the time-based retention to 3 days (259,200,000 milliseconds).
Understanding the Retention Mechanism:
Kafka doesn’t delete messages individually. Instead, it operates on log segments. A log segment is a file on disk containing a contiguous range of messages. Kafka periodically checks the size and age of these segments.
- Time-based deletion (
retention.ms): If a segment’s oldest message is older thanretention.ms, the entire segment is marked for deletion. - Size-based deletion (
retention.bytes): If the total size of all segments for a topic partition exceedsretention.bytes, Kafka starts deleting the oldest segments until the total size is below the limit.
You can set both time and size-based retention. Kafka will enforce whichever limit is reached first. For instance, if retention.ms=86400000 (1 day) and retention.bytes=1073741824 (1GB), old messages will be deleted after a day, or if the partition fills up to 1GB, whichever comes first.
Key Configuration Parameters:
retention.ms: The time in milliseconds to retain messages. Default is604800000(7 days). Set to-1to disable time-based retention.retention.bytes: The maximum size in bytes to retain messages for a partition. Default is-1(no size limit).log.segment.bytes: The maximum size of a single log segment file. Default is1073741824(1GB). This impacts how often Kafka rolls over to a new segment file.log.retention.check.interval.ms: How often Kafka checks for log segments to delete. Default is300000(5 minutes).
The "Per-Topic" Nuance:
The most powerful aspect is that these retention policies are configurable per topic. This means you can have a user_activity topic retaining data for 7 days and 10GB, while a system_logs topic might retain data for only 1 hour and 1GB. This fine-grained control is essential for managing storage costs and compliance requirements across different data streams.
The default retention.ms of 7 days is often a good starting point, but you’ll need to tune retention.bytes based on your expected message volume and disk capacity. If you set retention.ms to a very large value and retention.bytes to a small value, Kafka will aggressively delete old segments to stay under the byte limit, even if they are not yet expired by time. Conversely, a large retention.bytes and a small retention.ms means time will be the primary driver for deletion.
When Kafka deletes a log segment, it’s not just freeing up disk space; it’s fundamentally removing data from the event stream. This makes retention policies a critical aspect of data lifecycle management in any Kafka-based architecture.
The next challenge you’ll likely encounter is managing the "compaction" of log segments for topics where you want to retain the latest value for each key, regardless of time or size.