NATS JetStream’s "Max Age" and "Max Size" limits aren’t just arbitrary caps; they’re fundamental to managing the dynamic lifecycle and resource consumption of your message streams.
Let’s see JetStream in action, managing a stream with these limits. Imagine a scenario where we want to retain messages for at most 24 hours, and the total size of messages in the stream shouldn’t exceed 1 gigabyte.
{
"name": "sensor-data",
"subjects": ["sensors.>"],
"storage": "file",
"retention": "limits",
"max_age": 86400000000000, // 24 hours in nanoseconds
"max_size": 1073741824, // 1 GB in bytes
"discard": "old"
}
This configuration, when applied to a JetStream stream, tells the system: "Keep messages only as long as they are less than 24 hours old, and if the total data size ever hits 1GB, start discarding the oldest messages to make room." The discard: "old" policy is crucial here; it ensures that when either max_age or max_size is hit, JetStream will purge the oldest messages first.
The problem JetStream’s retention policies solve is the runaway growth of message data. Without them, a busy stream could consume all available disk space, leading to system instability and data loss. By setting max_age and max_size, you define a predictable, bounded storage footprint for your streams.
Internally, JetStream tracks the age and cumulative size of messages within each stream. When a new message arrives, it checks these limits. If adding the new message would violate max_size, or if the oldest message has exceeded max_age, JetStream triggers a cleanup process. This process iterates through the stream, removing messages from the beginning until both limits are satisfied. The storage: "file" option means this data is persisted to disk, making these limits critical for managing disk I/O and capacity.
The retention: "limits" setting is the key enabler for both max_age and max_size. If retention were set to interest, messages would only be purged when no consumers are actively acknowledging them. With limits, age and size become the primary drivers for message eviction, regardless of consumer activity.
What most people don’t realize is how max_age and max_size interact when both are configured. If a stream fills up to max_size before any messages reach max_age, JetStream will start discarding messages based on size. Conversely, if messages age out to max_age before the stream reaches max_size, JetStream will discard them based on age. The system continuously monitors both conditions and acts upon whichever limit is breached first, always adhering to the discard: "old" policy to maintain a consistent view of the "current" data.
The next logical step after managing stream limits is to consider how these limits affect message delivery and consumer behavior, particularly concerning message redelivery and acknowledgment timeouts.