NATS JetStream’s "replay" feature doesn’t just re-deliver messages; it can deliver them with their original timestamps, which is profoundly different from simply replaying them immediately.
Let’s see it in action. Imagine we have a JetStream stream named orders that’s configured to retain messages with a 1-hour TTL. We publish a message at 2023-10-27T10:00:00Z. Later, we want to replay this message.
Here’s how we’d configure a consumer to replay messages with their original timestamps:
nats context save my-nats-context
nats context edit my-nats-context --js-api-prefix "_api" # If using custom API prefix
# Create a stream (if it doesn't exist)
nats stream add orders --subjects "order.>" --max-age 1h
# Publish a message
nats pub order.new "payload1" --ack
# Wait a bit (e.g., 5 minutes) to simulate time passing
sleep 300
# Create a consumer that replays with original timing
nats consumer add orders replay-original --replay original --filter "order.new"
Now, when we consume from replay-original:
nats sub order.new --consumer replay-original --no-ack --stream orders
The message will be delivered with its original timestamp of 2023-10-27T10:00:00Z, even though we’re consuming it 5 minutes after that original publication time.
Contrast this with a consumer set to replay with instant timing (the default if not specified):
nats consumer add orders replay-instant --replay instant --filter "order.new"
nats sub order.new --consumer replay-instant --no-ack --stream orders
This replay-instant consumer would receive the message with a timestamp reflecting the moment it was replayed, not its original publication time.
The core problem JetStream replay solves is enabling systems to reprocess historical data without altering the perceived timeline of events. This is critical for financial systems, auditing, and any application where the exact timing of an event, not just its content, is paramount. When you replay with original timing, you’re essentially rewinding the clock for that specific message. The JetStream server stores the original publication timestamp and makes it available to consumers configured for original replay.
Internally, JetStream maintains metadata for each message, including its sequence number within the stream, the time it was persisted, and crucially, its original publication time. When a consumer is created with replay: original, JetStream uses this stored publication time as the delivery timestamp when replaying messages. For replay: instant, it uses the current server time at the moment of replay.
The filter option on consumers is powerful here. You can replay specific subsets of messages based on subject filters. For example, if you have a stream with order.created and order.shipped subjects, you can create a consumer to only replay order.created messages with their original timing.
The replay setting is a property of the consumer, not the stream. This means you can have multiple consumers on the same stream, each with a different replay strategy. One might replay instantly for immediate re-processing, while another replays with original timing for historical analysis.
A subtle but powerful aspect is how JetStream handles message redelivery after a consumer has processed a message. If a consumer is configured for original replay and experiences an error, when it restarts and re-fetches messages, it will still receive them with their original timestamps, ensuring consistent reprocessing against the historical event timeline.
The next logical step when dealing with replay is understanding how to manage the lifecycle of these replay consumers, particularly when dealing with large historical datasets and ensuring efficient consumption without overwhelming downstream systems.