The NATS JetStream server stopped acknowledging messages that consumers were pulling, causing them to be redelivered indefinitely.

This happens because the JetStream server believes the message was never successfully delivered to the consumer, even though the consumer did receive it. The server’s internal state is out of sync with reality.

The most common culprit is a network partition or dropped connection between the JetStream server and the consumer. When the consumer acknowledges a message, that ack is sent back to the server. If that ack packet gets lost, the server never sees it and assumes the message is still outstanding.

Diagnosis:

First, check your NATS server logs for any errors related to consumer acknowledgements or message redelivery. Look for patterns like consumer XXX not found or message YYY still unacknowledged.

Next, inspect the consumer’s state on the NATS server. Use the nats consumer info <stream_name> <consumer_name> command. Pay close attention to the delivered and acknowledged counts. If delivered is significantly higher than acknowledged, or if redelivered is climbing rapidly, this confirms the issue.

nats consumer info my_stream my_consumer

Common Cause 1: Transient Network Issues

  • Diagnosis: Network packet loss between the NATS server and the consumer. Check network monitoring tools or ping and traceroute from the consumer to the NATS server.
  • Fix: This is often self-healing. If persistent, review network infrastructure. For a quick reset on the consumer side, restart the consumer application. This forces it to re-establish its connection and potentially re-sync its last acknowledged sequence.
  • Why it works: A fresh connection can clear stale state on the consumer and allow it to resend its last known acknowledged sequence number to the server.

Common Cause 2: Consumer Application Crashes/Restarts

  • Diagnosis: If your consumer application crashes and restarts, it might not have properly reported its last acknowledged message sequence before going down.
  • Fix: Ensure your consumer application has robust error handling and graceful shutdown procedures. Before exiting, it should explicitly acknowledge any messages it has processed. If the crash was sudden, you might need to manually reset the consumer’s state on the NATS server.
  • Example Fix (Manual Reset): If you know the consumer was processing messages up to sequence 12345, you can try to reset its delivered sequence. Be very careful with this command, as it can lead to duplicate processing if not used correctly.
    nats consumer reset my_stream my_consumer --deliver-seq 12345
    
    This tells the consumer to start delivering messages from sequence 12345. The server will then expect acknowledgements for messages after this.
  • Why it works: This manually aligns the consumer’s internal state with what the server should have seen acknowledged, preventing it from re-delivering messages it already processed.

Common Cause 3: High Consumer Load / Slow Processing

  • Diagnosis: The consumer is receiving messages faster than it can process and acknowledge them. This can lead to a backlog and the server eventually timing out outstanding messages.
  • Fix: Increase the number of consumer instances or threads, optimize your message processing logic, or increase the ack_wait duration on the consumer configuration.
  • Example Fix (Increase ack_wait):
    nats consumer update my_stream my_consumer --ack-wait 60s
    
    This gives the consumer 60 seconds to acknowledge a message after it’s delivered, reducing the chance of premature redelivery due to processing delays.
  • Why it works: It provides more breathing room for the consumer to complete its work and send acknowledgements before the server considers the message unacknowledged.

Common Cause 4: NATS Server Resource Starvation

  • Diagnosis: The NATS server itself is under heavy load (CPU, memory, disk I/O) and is slow to respond to acknowledgements or to update its internal state.
  • Fix: Scale up your NATS server resources, optimize its configuration, or reduce the overall message throughput to the server.
  • Why it works: Ensures the NATS server can perform its essential duties of tracking message delivery and acknowledgements in a timely manner.

Common Cause 5: Consumer Configuration Drift

  • Diagnosis: The consumer was created with certain parameters (e.g., ack_wait, max_deliver), but these were changed on the server side without the consumer being aware, or vice-versa.
  • Fix: Recreate the consumer with the desired configuration.
    nats consumer delete my_stream my_consumer
    nats consumer add my_stream my_consumer --ack-wait 30s --deliver-policy all --sample 100
    # ... other desired options
    
  • Why it works: A fresh consumer instance starts with a clean slate and the exact configuration you intend, resolving any discrepancies.

Common Cause 6: Message Payload Corruption (Rare)

  • Diagnosis: In extremely rare cases, a malformed message payload could cause issues with the consumer’s processing logic, preventing it from successfully acknowledging the message.
  • Fix: Implement checksums or data validation within your message processing. If a message consistently fails processing and acknowledgement, you might need to inspect its contents directly (if possible) or configure the consumer to discard repeatedly failing messages after a certain number of redeliveries (using max_deliver).
  • Example Fix (Set max_deliver):
    nats consumer update my_stream my_consumer --max-deliver 10
    
    This will cause the consumer to stop redelivering a message after it has been delivered 10 times, preventing an infinite loop.
  • Why it works: Limits the impact of a single problematic message and allows other messages to proceed.

After resolving the underlying cause, you might see a brief spike in redelivered messages as the system catches up, but the "No Message Found" error should disappear. The next error you’ll likely encounter is related to duplicate message processing if your consumer logic isn’t idempotent.

Want structured learning?

Take the full Nats course →