The NATS JetStream API is failing because the server is unable to fulfill requests due to a fundamental misconfiguration of its storage or stream definitions.
Insufficient Storage Space
Diagnosis: Check disk space on the NATS server hosting JetStream.
df -h /var/lib/nats
Cause: JetStream requires disk space for its stream data and metadata. If the underlying filesystem is full, JetStream cannot write new messages or update stream state, leading to API errors.
Fix:
Free up disk space or expand the storage volume. For example, if the /var/lib/nats directory is on a full partition:
# Example: If you have old log files to clear
sudo rm /var/log/old_service.log.*
Why it works: This action frees up blocks on the storage device, allowing the NATS server to allocate space for JetStream’s operations.
Incorrect Stream Configuration (Max Age/Size)
Diagnosis: Inspect the problematic stream’s configuration via the NATS CLI or API.
nats stream info <stream_name>
Look for max_age or max_size settings that are too restrictive.
Cause: Streams have retention policies. If max_age is set to a short duration and messages are older than that, or if max_size is reached, JetStream will start rejecting new publishes or consumers might not see older messages.
Fix:
Adjust the stream’s retention policy to be more permissive. For example, to increase max_age to 7 days and max_size to 10GB:
nats stream update <stream_name> --max-age 7d --max-size 10Gi
Why it works: This expands the window for message retention or the total data storage, preventing premature data eviction or rejection due to policy limits.
Stream Purged or Deleted
Diagnosis: Check if the stream exists.
nats stream list
If the stream name is not present, it has been deleted.
Cause: The stream might have been accidentally or intentionally purged (all messages removed) or the entire stream definition deleted. API calls targeting a non-existent stream will fail.
Fix: Recreate the stream with the desired configuration.
nats stream add <stream_name> --subjects "my.subject.>" --storage file --replicas 3
Why it works: This re-establishes the stream object within JetStream, allowing messages to be published and consumed again according to the new definition.
Consumer Configuration Issues (Deliver Policy/Start Sequence)
Diagnosis: Examine the specific consumer in question.
nats consumer info <stream_name> <consumer_name>
Pay attention to deliver_policy and deliver_subject.
Cause: Consumers can be configured to start from the beginning (DeliverAll), the last stored message (DeliverLast), or a specific sequence number (DeliverNew). If a consumer is misconfigured to start from a sequence number that no longer exists due to stream retention, it will error. Similarly, if the deliver_subject is not valid or reachable, messages won’t be delivered.
Fix:
Update the consumer’s delivery policy or starting point. If the stream has aged out messages and the consumer was set to DeliverNew with an old sequence:
nats consumer update <stream_name> <consumer_name> --deliver-policy all
If the deliver_subject is the issue, ensure it’s a valid NATS subject and the consumer has permissions to publish to it.
Why it works: This aligns the consumer’s starting point with the available data in the stream or corrects the delivery endpoint, allowing the consumption process to proceed.
JetStream Server Not Running or Unhealthy
Diagnosis: Check the status of the NATS server process.
sudo systemctl status nats-server
Also, check the NATS server logs for any JetStream-specific errors.
Cause: The NATS server process itself might have crashed or failed to start, or the JetStream component within the server might be in an unhealthy state due to internal errors. Without a running JetStream server, all API requests will fail.
Fix: Restart the NATS server.
sudo systemctl restart nats-server
If the issue persists, examine the NATS server configuration file (nats-server.conf) for JetStream-related settings and ensure they are correct, especially the jetstream block.
Why it works: Restarting the service ensures that the JetStream component is re-initialized correctly. Correcting configuration issues prevents the JetStream component from entering an unhealthy state upon startup.
Insufficient File Descriptors
Diagnosis: Check the open file descriptor limits for the NATS server process.
sudo lsof -p $(pgrep -f nats-server) | wc -l
ulimit -n
Compare the output of lsof to the ulimit -n value.
Cause: JetStream, especially with many streams and consumers, can open a large number of file descriptors for its data files and network connections. If the system’s limit on open file descriptors per process is too low, the NATS server will fail to open new files or establish new connections, leading to API errors.
Fix:
Increase the system’s or user’s file descriptor limit. This is typically done by editing /etc/security/limits.conf or systemd service files. For a system-wide change:
Add these lines to /etc/security/limits.conf:
* soft nofile 65536
* hard nofile 65536
Then, restart the NATS server.
Why it works: This raises the maximum number of files and network sockets that the NATS server process can have open concurrently, allowing JetStream to manage its internal state and connections without hitting resource limits.
The next error you’ll likely encounter is related to TLS certificate validation if you’re using JetStream over HTTPS and haven’t configured it correctly.