Kafka Quotas allow you to limit the amount of data producers and consumers can send or retrieve from Kafka.
Let’s see it in action. Imagine you have a critical Kafka cluster and you want to prevent a specific application, let’s say order-processor, from overwhelming it. You can set quotas on this client’s ingress (data it sends) and egress (data it receives).
Here’s how you’d configure it in server.properties:
# Producer Quota
producer.quota.bytes.per.second.default = 104857600 # 100MB/s
producer.quota.messages.per.second.default = 100000
# Consumer Quota
consumer.quota.bytes.per.second.default = 52428800 # 50MB/s
consumer.quota.messages.per.second.default = 50000
These are global defaults. To apply them to a specific client, you’d use client identification. Kafka can identify clients by:
- User Principal: If you have authentication enabled (e.g., SASL), you can associate quotas with specific authenticated users.
- Client ID: Producers and consumers can set a
client.idproperty in their configuration. This is a string that uniquely identifies the application instance. This is generally the most flexible for throttling specific applications.
Let’s say our order-processor producer has the client.id set to order-processor-app-1. You can then configure a user-specific quota for that client ID.
If using Kafka’s authorization mechanism (ACLs), you’d typically define quotas based on the principal that owns the client ID. For example, if your order-processor connects with a SASL user app_user, you’d set the quota for User:app_user.
You can also set quotas based on IP address, though this is less common for application-level throttling.
The actual mechanism works by Kafka brokers monitoring the data transfer rates for each client. When a client exceeds its configured quota, the broker will start throttling its requests. For producers, this means send() calls will block or return errors (depending on configuration and client retries). For consumers, fetch requests will be delayed, effectively slowing down their consumption rate.
The key levers you control are:
producer.quota.bytes.per.second.<user or client_id>: Limits the total bytes sent by a producer per second.producer.quota.messages.per.second.<user or client_id>: Limits the number of messages sent by a producer per second.consumer.quota.bytes.per.second.<user or client_id>: Limits the total bytes fetched by a consumer per second.consumer.quota.messages.per.second.<user or client_id>: Limits the number of messages fetched by a consumer per second.
You can also set default quotas that apply to all clients unless overridden by specific configurations. The default values are often set to be very high to avoid impacting normal operations.
The most surprising thing about Kafka Quotas is that they are enforced on a per-broker basis. This means if you have a multi-broker cluster, each broker independently enforces the quota for the clients connecting to it. If a client connects to multiple brokers simultaneously, its total throughput will be the sum of the throttled rates across all brokers it’s connected to, potentially leading to a more complex throttling behavior than a single, unified rate limit. This is crucial to understand for accurate capacity planning and troubleshooting.
The next challenge you’ll likely face is how to dynamically adjust these quotas without restarting brokers.