NATS streams don’t actually "retain" messages; they acknowledge them, and the actual message storage and purging is handled by the underlying stream configuration.
Let’s see NATS retention in action. Imagine we have a stream my-stream that’s configured to hold onto messages for a maximum of 1 minute, or a maximum of 100 messages, whichever comes first.
nats stream add my-stream --storage file --retention limits --max-age 1m --max-msgs 100
When a message arrives for a subject this stream is listening to, NATS will store it. If the stream reaches 100 messages, the oldest ones are purged. If a message has been in the stream for more than 1 minute, it’s purged, even if there are fewer than 100 messages. This ensures that the stream doesn’t grow indefinitely.
The core problem NATS retention solves is preventing unbounded storage growth and ensuring that stale data is automatically cleaned up. This is crucial for systems that might experience traffic spikes or have long-running processes that don’t need access to historical data beyond a certain point.
Internally, when a message is added to a stream, NATS checks the current message count and the age of the oldest message against the max-msgs and max-age limits. If either limit is exceeded, NATS initiates a purge operation. For file storage, this involves scanning the data files and removing segments that contain only messages older than max-age or are no longer needed due to max-msgs limits. For memory storage, it’s a simpler in-memory cleanup.
You control retention primarily through two parameters when creating or updating a stream: max-age and max-msgs. max-age takes a duration (e.g., 1h, 30m, 5s), and max-msgs takes an integer count. You can also combine these with max-bytes to limit storage by size. The retention policy itself can be limits (the default, using max-age, max-msgs, max-bytes), interest (purges messages that have no active consumers interested in them), or workqueue (a special type for work queue semantics where messages are acknowledged and purged upon successful processing).
The interest retention policy is particularly powerful for scenarios where you want to ensure that messages are only kept as long as they are actively being consumed. If a message has been published but no consumer has yet acknowledged it, and there are no consumers currently listening for that subject, NATS can eventually purge that message. This is different from limits because it’s not based on a fixed time or count but on the dynamic state of consumer interest.
When you set retention interest, NATS keeps track of which messages are "delivered" (sent to a consumer) and which are "acknowledged" (by a consumer). A message is considered eligible for purging if it has been delivered to all consumers that ever expressed interest in it, and those consumers have subsequently acknowledged it, or if no consumer has ever expressed interest in it. NATS periodically checks for such eligible messages and purges them. This is managed internally by tracking acknowledgments and consumer state.
The most surprising thing about NATS retention, especially the interest policy, is how it manages state without requiring explicit consumer heartbeats or complex client-side logic for message expiry. NATS itself becomes the arbiter of "staleness" based on consumption patterns, which simplifies client implementations significantly.
The next concept to explore is how to manage message ordering and deduplication within streams, especially when dealing with multiple publishers or potential network partitions.