NATS JetStream consumers are the primary way your applications interact with messages, but understanding the difference between "pull" and "push" consumers is crucial for efficient message processing.
Let’s see a push consumer in action. Imagine a simple NATS JetStream stream named orders. We want to process new orders as they arrive.
nats context create my-nats --server nats://localhost:4222
nats context use my-nats
# Create a stream
nats stream add orders --subjects orders.new
# Publish a message
nats pub orders.new "{\"order_id\": 123, \"item\": \"widget\"}"
# Create a push consumer
nats consumer add orders my-push-consumer --ack-explicit --deliver-policy instant --filter-subject orders.new
Now, let’s simulate a client application that listens to this push consumer. In a real application, this would be code using a NATS client library. For demonstration, we’ll use the nats CLI to receive messages.
nats sub orders.new --consumer my-push-consumer --stream orders --no-ack
When you run the nats sub command, the NATS server immediately starts pushing messages to your subscriber as soon as they are published to the orders.new subject and match the consumer’s filter. The server actively manages the delivery, and your client simply receives them.
Now, consider a pull consumer for the same orders stream. This time, your application dictates when it wants to receive messages.
# Create a pull consumer
nats consumer add orders my-pull-consumer --ack-explicit --deliver-policy instant --filter-subject orders.new --pull
To interact with this pull consumer, your application would use a NATS client library to explicitly request messages. Again, we’ll use the nats CLI for demonstration.
# Request messages from the pull consumer
nats pull orders.new --consumer my-pull-consumer --stream orders --count 1
When you run nats pull, you are asking the NATS server for up to count messages. The server waits until you request them. This gives your application complete control over the flow, allowing it to process messages at its own pace, batch them, or even implement sophisticated retry logic based on its current capacity.
The core problem JetStream solves is reliable message delivery in a distributed system, moving beyond the "fire and forget" nature of traditional NATS subjects. JetStream adds persistence and consumer management. Push consumers are ideal when you want real-time processing and can handle a continuous stream of messages. The NATS server acts as the active distributor, pushing messages to consumers as they become available. This is efficient for high-throughput scenarios where latency is critical, and your consumers are always ready to receive.
Pull consumers, on the other hand, are designed for scenarios where your application needs more control over message consumption. This could be due to processing constraints, the need for batching, or the desire to implement custom flow control. Your application initiates the request for messages, and the NATS server responds with what’s available. This model decouples message production from consumption, allowing for more resilient and adaptable client applications. You can also use pull consumers for scheduled or on-demand message processing.
The ack-explicit option, used in both examples, is fundamental. It means your application must acknowledge receipt and successful processing of a message before JetStream considers it done. If your application crashes before acknowledging, JetStream will redeliver the message to another consumer (or the same one if it restarts). This is the bedrock of JetStream’s "at-least-once" delivery guarantee. Without ack-explicit, JetStream would acknowledge messages automatically upon delivery, leading to potential data loss if your application failed mid-processing.
A common pitfall with push consumers is that if your application can’t keep up with the incoming message rate, it can overwhelm its own buffers, leading to dropped messages or increased latency. The NATS server tries to manage this with flow control, but ultimately, the consumer’s processing speed is the bottleneck. For pull consumers, the "pitfall" is reversed: if your application doesn’t pull messages frequently enough, they can accumulate in JetStream, potentially leading to higher memory usage on the server side for buffered messages and increased latency when consumption eventually resumes.
The next concept you’ll likely explore is how to manage message acknowledgments and redelivery policies for these consumers, especially when failures occur.