Articles

Fix NATS Duplicate Message ID Deduplication (2026)

April 18, 2026 5 min read

The NATS JetStream server failed to acknowledge messages to a publisher because it couldn’t verify their unique message_id values, leading to publishers re-sending messages.

This usually happens when the JetStream server’s memory of seen message_ids gets out of sync with the actual stream state, often due to restarts or clock drift.

Cause 1: Clock Skew Between Publisher and Server

Diagnosis: Check the system clocks on your publisher instances and your NATS server instances.

ssh publisher-host "date"
ssh nats-server-host "date"

Look for differences greater than a few seconds.

Fix: Synchronize clocks using NTP. On your publisher and NATS server hosts:

sudo apt-get update && sudo apt-get install ntp -y
sudo systemctl enable ntp
sudo systemctl start ntp

This ensures consistent time-based deduplication checks.

Why it works: JetStream uses timestamps and message_ids (which often incorporate timestamps) to deduplicate. If clocks are skewed, a message might appear older or newer than it actually is relative to the server’s understanding, leading to incorrect deduplication decisions.

Cause 2: JetStream Stream Configuration - max_age Too Short

Diagnosis: Examine your JetStream stream configuration. If max_age is set to a value shorter than the potential latency for message acknowledgment, deduplication information might be purged before a message can be fully processed and acknowledged by all consumers.

nats stream info your_stream_name

Look for the max_age field.

Fix: Increase max_age to a value comfortably longer than your expected message processing and acknowledgment cycle, for example, to 24h.

nats stream update your_stream_name --max-age 24h

This gives the JetStream server more time to retain deduplication information for messages.

Why it works: The deduplication buffer is tied to the stream’s retention policy. If max_age is too short, the server discards the message_id lookup data too early, causing it to believe a message it has already processed is new.

Cause 3: JetStream Stream Configuration - dedupe_window Too Small

Diagnosis: Check the dedupe_window setting for your stream. This parameter defines how long JetStream keeps message_ids in memory for deduplication checks after a message has been acknowledged. If this window is too small, messages sent in quick succession might fall outside the window before the server can confirm their successful processing.

nats stream info your_stream_name

Look for the dedupe_window field.

Fix: Increase dedupe_window. A common starting point is 1m or 5m, depending on your message send rate.

nats stream update your_stream_name --dedupe-window 5m

This extends the period during which the server actively checks for duplicate message_ids.

Why it works: The dedupe_window directly controls the lifespan of the deduplication lookup table. A larger window ensures that even if there are slight delays in acknowledgment, the server still has the message_id in its history to detect duplicates.

Cause 4: Publisher Not Setting message_id Consistently

Diagnosis: Verify that your publisher code is consistently generating and setting a unique message_id for every message it sends. This ID should be truly unique for the duration of the dedupe_window. A common mistake is reusing IDs or generating them based on non-unique local state. Check your publisher logs for errors related to message_id generation or for instances where it’s not being set.

Fix: Implement a robust message_id generation strategy in your publisher. Using a UUID v4 or a combination of a unique publisher instance ID and a monotonically increasing sequence number is recommended. Ensure this ID is correctly populated in the NATS message headers. Example (Go):

import "github.com/google/uuid"
// ...
msgID := uuid.New().String()
msg := nats.Msg{
    Subject: "my.subject",
    Data:    []byte("my payload"),
    Header: map[string][]string{
        "Nats-Msg-Id": {msgID},
    },
}
// ... publish msg

This guarantees that each logical message has a distinct identifier for deduplication.

Why it works: The entire deduplication mechanism relies on the uniqueness of the message_id. If the publisher fails to provide unique IDs, JetStream cannot differentiate between genuine duplicates and distinct messages.

Cause 5: JetStream Server Restart During Message In-Flight

Diagnosis: Review NATS server logs for any restarts that occurred during periods of high message throughput. If a server restarts, it loses its in-memory deduplication state. If a publisher is re-sending a message that was already processed but not yet fully acknowledged by JetStream before the restart, the new server instance won’t recognize it as a duplicate.

Fix: There’s no direct "fix" for a restart causing this, as state loss is inherent. However, a robust strategy involves:

  1. Graceful Shutdowns: Ensure your NATS servers are configured for graceful shutdowns, allowing in-flight acknowledgments to complete.
  2. Publisher Retries: Publishers should implement their own retry logic with exponential backoff and jitter, and always re-send with the same message_id after a timeout or error.
  3. Durable Consumers: Ensure consumers are durable so they can resume processing from where they left off after a server restart, which helps clear the backlog and re-establish acknowledgment state.

Why it works: While you can’t prevent state loss on restart, you can mitigate its effects by ensuring publishers consistently re-send with the same message_id and that consumers are resilient, allowing the system to eventually reconcile duplicate states.

Cause 6: High Volume of Messages Exceeding dedupe_window Capacity

Diagnosis: Monitor your JetStream stream’s ingress and egress rates. If the rate of incoming messages and the rate at which they are acknowledged exceeds the capacity of the dedupe_window to hold message_ids, you can experience "window overflow." This isn’t an error per se, but a system saturation point where the deduplication cache becomes ineffective. Check JetStream metrics for jetstream.api.msg.store.ingress and jetstream.api.msg.store.egress rates.

Fix:

  1. Increase dedupe_window: If your message rates are consistently high, you may need a significantly larger dedupe_window.
    nats stream update your_stream_name --dedupe-window 1h
  2. Scale NATS JetStream: If increasing the window isn’t feasible due to memory constraints, you might need to scale your NATS JetStream cluster to distribute the load.
  3. Optimize Publisher: Reduce the message sending rate if possible, or batch messages to send fewer, larger messages.

Why it works: A larger dedupe_window allows the JetStream server to keep track of more message_ids for a longer duration, accommodating higher throughput without prematurely discarding relevant deduplication data. Scaling distributes the load, meaning each server instance handles fewer messages and can manage its deduplication state more effectively.

The next error you’ll likely encounter after fixing deduplication issues is a ConsumerNotBound error if consumers haven’t been properly re-attached to the stream after a server restart or if their state became corrupted.

Want structured learning?

Take the full Nats course →