Messaging systems are fundamentally about decoupling producers from consumers, but when producers get too enthusiastic, consumers can drown. This is where rate limiting and backpressure become your best friends.
Let’s see this in action with a conceptual Kafka producer and consumer. Imagine a producer churning out messages to a topic user_events faster than a single consumer can process them.
// Producer sending messages
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
for (int i = 0; i < 1000000; i++) {
String msg = "User event " + i;
producer.send(new ProducerRecord<>("user_events", Integer.toString(i), msg));
// Without rate limiting, this loop can overwhelm the broker or consumers
}
producer.close();
// Consumer receiving messages
Properties consumerProps = new Properties();
consumerProps.put("bootstrap.servers", "localhost:9092");
consumerProps.put("group.id", "user_event_processor");
consumerProps.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
consumerProps.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
consumerProps.put("enable.auto.commit", "false"); // Crucial for backpressure
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(consumerProps);
consumer.subscribe(Arrays.asList("user_events"));
while (true) {
ConsumerRecords<String, Integer> records = consumer.poll(Duration.ofMillis(100));
if (!records.isEmpty()) {
for (ConsumerRecord<String, Integer> record : records) {
// Process the message
System.out.println("Received: " + record.value());
// If processing is slow, we don't commit offsets, causing a backlog
}
consumer.commitSync(); // Only commit after successful processing
}
// If poll returns few records or none, it implies backpressure is working
}
The core problem rate limiting and backpressure solve is preventing a fast producer from overwhelming a slow consumer or the messaging system itself. Without them, you get dropped messages, increased latency, and potential system instability. Rate limiting is about controlling the producer’s output, while backpressure is about the consumer signaling its inability to keep up, which then propagates upstream.
In Kafka, rate limiting is often handled at the client level or by the broker. You can configure max.in.flight.requests.per.connection on the producer to limit how many requests are outstanding. For broker-level rate limiting, you can set limits. properties in server.properties like limits.producer_rate_limit.bytes_per_second and limits.consumer_rate_limit.bytes_per_second. These are absolute caps.
Backpressure in Kafka is more implicit and relies on consumer behavior and broker buffering. The enable.auto.commit setting on the consumer is key. When set to false, the consumer is responsible for explicitly committing offsets after it has successfully processed a batch of records. If the consumer is slow, it won’t commit frequently. Kafka brokers, by default, retain data for a configurable period (e.g., log.retention.hours=168). If a consumer falls behind, it will eventually hit the oldest available offset for its partition. The broker won’t serve data beyond that point until the consumer catches up and commits. This naturally throttles the consumer’s fetch requests.
Beyond Kafka’s built-in mechanisms, you might implement explicit rate limiting in your producer application logic. This could involve a token bucket algorithm or simply a Thread.sleep() after sending a certain number of messages within a time window. For backpressure, a common pattern is to use bounded queues between your message polling loop and your processing logic. If the queue is full, you stop polling for new messages until space is available.
The most surprising truth about backpressure is that it’s not always about actively stopping data flow, but often about allowing it to naturally slow down by not immediately acknowledging or storing processed data. When a Kafka consumer with enable.auto.commit=false fails to process records, it simply doesn’t advance its offset. The next poll() will return the same uncommitted records. If this continues, the consumer effectively stops consuming. The broker, seeing no progress, will continue to serve these same records until they are processed or expire from retention. This is a powerful, passive form of backpressure.
The next logical step after mastering rate limiting and backpressure is understanding how to handle failures within the processing pipeline itself, ensuring idempotency and exactly-once processing semantics.