Fluent Bit’s output plugins are designed to send logs to a single destination. When you need to distribute logs across multiple upstream hosts, you’re essentially fighting against the default behavior of these plugins.
Here’s Fluent Bit sending logs to a single Elasticsearch cluster:
[OUTPUT]
Name es
Match *
Host elasticsearch.example.com
Port 9200
Logstash On
Replace_Dot On
Retry_Limit 5
Now, imagine you have es-node-1.example.com, es-node-2.example.com, and es-node-3.example.com, all running Elasticsearch. You want Fluent Bit to send logs to them in a round-robin fashion. The most direct way to achieve this is by leveraging the Host parameter’s ability to accept a comma-separated list of hosts.
[OUTPUT]
Name es
Match *
Host es-node-1.example.com,es-node-2.example.com,es-node-3.example.com
Port 9200
Logstash On
Replace_Dot On
Retry_Limit 5
When Fluent Bit encounters multiple hosts in the Host field, it automatically engages a round-robin load-balancing mechanism. It picks the first host in the list for the initial connection. Subsequent connections will cycle through the remaining hosts in the order they appear. If a connection to a host fails, Fluent Bit will attempt to connect to the next host in the list, respecting the Retry_Limit for that specific host before moving on. Once it exhausts the list and retries have failed for all, it will eventually cycle back to the beginning of the list for new connection attempts.
This round-robin behavior is built into the output plugin’s connection logic. It’s not a separate load-balancing component you need to configure. The plugin itself manages the distribution.
What happens under the hood is that Fluent Bit maintains an internal index for the list of hosts. When it needs to send a batch of records, it selects the host at the current index. After a successful send or a failed attempt that exhausts retries for that host, it increments the index (wrapping around to 0 if it reaches the end of the list). This ensures that over time, the load is distributed across all specified upstream hosts.
The Retry_Limit parameter is crucial here. If set to 0, Fluent Bit will not retry a failed connection to a specific host and will immediately move to the next one in the list. Setting it to a value greater than 0 means Fluent Bit will attempt to reconnect to the same host multiple times before giving up on that host for the current batch and moving on. This can be beneficial if you have transient network issues or a host is temporarily overloaded.
Consider the case where you have a mix of healthy and unhealthy upstream hosts. Fluent Bit’s round-robin will continue to attempt connections to unhealthy hosts until their Retry_Limit is exhausted. It won’t dynamically detect and remove unhealthy hosts from the rotation. This means you might see a temporary skew in load towards the healthy hosts if one or more are consistently failing.
To achieve more sophisticated load balancing, such as weighted distribution or health checks, you would typically place an external load balancer (like HAProxy, Nginx, or a cloud provider’s load balancer) in front of your upstream hosts. Fluent Bit would then be configured to send logs to the single IP address or hostname of that external load balancer.
The most common misconception is that you need to configure a separate load-balancing input or output plugin. The capability is inherent to how the Host parameter handles multiple values in many standard output plugins.
The next challenge you’ll likely face is ensuring that your upstream hosts have sufficient capacity to handle the aggregated log volume and that your log routing logic within Fluent Bit (using Match directives) is precise enough to send the right logs to the right destinations.