Fluent Bit’s Throttle filter allows you to control the rate of logs being processed, preventing downstream systems from being overwhelmed.
Here’s how it works in practice. Imagine you have a noisy application generating tons of logs, and you want to send them to a central logging system but don’t want to blow out your ingest budget or overwhelm your Elasticsearch cluster. You can use the Throttle filter to cap the rate.
Consider this Fluent Bit configuration:
[SERVICE]
Flush 5
Daemon off
Log_Level info
[INPUT]
Name tail
Path /var/log/myapp.log
Tag myapp.log
[FILTER]
Name throttle
Match myapp.log
Rate 1000 # Maximum 1000 records per second
Window 10 # Check rate over a 10-second window
Print_Status true # Optional: Log status of throttling
[OUTPUT]
Name es
Match myapp.log
Host localhost
Port 9200
Logstash_Format on
Replace_Dots on
In this setup, Fluent Bit will tail /var/log/myapp.log. Any logs from this source will be tagged myapp.log. The throttle filter, matching myapp.log, will then kick in. It’s configured to allow a maximum of 1000 records per second (Rate 1000). This rate is evaluated over a 10-second window (Window 10). If the rate exceeds 1000 records per second averaged over the last 10 seconds, Fluent Bit will start dropping logs until the average rate falls back below the threshold. The Print_Status true option will cause Fluent Bit to log messages indicating when throttling is active and when it has eased. Finally, the filtered logs are sent to an Elasticsearch instance running on localhost:9200.
The core problem this solves is preventing upstream systems from drowning downstream consumers. Think of it like a water pressure regulator for your logs. Without it, a sudden spike in log volume could cause your log aggregation service to crash, your database to become unresponsive, or your cloud provider to hit you with unexpected overage charges. The Throttle filter acts as a buffer, smoothing out these spikes by selectively dropping excess logs. It doesn’t try to magically process more data; it gracefully degrades by shedding load.
Internally, the Throttle filter maintains a sliding window of timestamps for incoming records. For each incoming record, it checks how many records have arrived within the configured Window. If the count exceeds the Rate multiplied by the Window, it means the average rate over that period has been too high, and the record is dropped. The Print_Status option is particularly useful for debugging; it will output messages like [throttle] rate limit exceeded, dropping records when it’s actively dropping logs, and [throttle] rate limit eased when the backlog clears.
The Rate parameter is the absolute maximum number of records per second you want to allow on average over the Window. The Window parameter defines the time period over which this average is calculated. A larger Window leads to smoother throttling but might allow brief, higher spikes before it kicks in. A smaller Window reacts faster to spikes but can be more sensitive to transient bursts that might not actually be problematic in the long run.
One subtlety many overlook is that the Rate is measured in records per second, not bytes per second. If you have very large log messages, you could hit your target record rate quickly even if you’re not sending a massive amount of data. Conversely, if your logs are very small, you could send gigabytes per second and still be under your record rate limit. You need to balance the Rate with your understanding of your typical log message size and the capacity of your downstream systems.
The next logical step after controlling the rate of logs is to control the content of the logs that are allowed through, for example, using the Grep filter to only pass logs matching specific patterns.