Fluent Bit’s configuration is a labyrinth of directives, and migrating from an older version can feel like navigating that labyrinth blindfolded. The surprising truth is that the core parsing logic hasn’t changed fundamentally, but the default behaviors and available plugins have evolved significantly, often in ways that silently break existing setups.
Let’s see Fluent Bit in action, processing logs with a modern configuration. Imagine this fluent-bit.conf:
[SERVICE]
Flush 5
Daemon Off
Log_Level info
Parsers_File parsers.conf
[INPUT]
Name tail
Path /var/log/myapp/*.log
Parser myapp_log
Tag myapp.logs
[FILTER]
Name modify
Match myapp.logs
Rename log message
Add processed_by fluentbit
[OUTPUT]
Name stdout
Match myapp.logs
Format json
This configuration tells Fluent Bit to:
- Flush buffered logs every 5 seconds.
- Run in the foreground (useful for debugging).
- Log at the
infolevel. - Use a custom parser defined in
parsers.conf.
The tail input plugin watches /var/log/myapp/*.log files, applies the myapp_log parser to each line, and tags the resulting records with myapp.logs.
The modify filter plugin, matching records tagged myapp.logs, renames the field log to message and adds a new field processed_by with the value fluentbit.
Finally, the stdout output plugin, also matching myapp.logs, prints the processed records to standard output in JSON format.
The problem Fluent Bit solves is efficiently collecting, transforming, and routing large volumes of log data from diverse sources to various destinations. Internally, it operates as a lightweight, high-performance log processor. It reads logs from inputs, applies filters for enrichment or modification, and then sends them to outputs. The key is its multi-threaded architecture and efficient buffer management, allowing it to handle thousands of events per second on modest hardware.
When upgrading Fluent Bit, especially from versions prior to 1.8, you’ll encounter several common migration pitfalls related to configuration directives.
One significant change is the deprecation and eventual removal of the time_key and time_format directives within input plugins like tail. In older versions, you might have had:
[INPUT]
Name tail
Path /var/log/legacy.log
Tag legacy
Time_Key timestamp
Time_Format %Y-%m-%d %H:%M:%S
This explicitly told Fluent Bit which field contained the timestamp and how to parse it. The modern approach consolidates this logic into the Parsers directive, leveraging the regex parser’s ability to capture and format timestamps. The equivalent in a modern configuration would look like this, assuming legacy_log is a parser defined elsewhere that extracts timestamp:
[INPUT]
Name tail
Path /var/log/legacy.log
Tag legacy
Parser legacy_log
And in parsers.conf:
[PARSER]
Name legacy_log
Format regex
Regex ^(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?<message>.*)$
Time_Key timestamp
Time_Format %Y-%m-%d %H:%M:%S
This separation of concerns makes configurations cleaner and more maintainable. The parser now defines how to extract fields and how to interpret the timestamp, rather than the input plugin trying to do both.
Another common area of change is how HTTP input and output plugins handle TLS configuration. Older versions used tls (on/off) and tls_ca_file, tls_cert_file, tls_key_file. Modern Fluent Bit uses a more granular [TLS] section within the respective plugin configuration, or a global [TLS] section for shared settings. For example, an older HTTP output might have:
[OUTPUT]
Name http
Match *
Host your.log.server
Port 443
tls On
tls_ca_file /etc/fluent-bit/ca.crt
tls_cert_file /etc/fluent-bit/client.crt
tls_key_file /etc/fluent-bit/client.key
This would be migrated to:
[OUTPUT]
Name http
Match *
Host your.log.server
Port 443
[TLS]
Enabled On
CA_File /etc/fluent-bit/ca.crt
Cert_File /etc/fluent-bit/client.crt
Key_File /etc/fluent-bit/client.key
The directive names have changed (tls to Enabled, tls_ca_file to CA_File, etc.), and the structure is now nested, which is more organized.
The mem_buf_limit directive, which controls the maximum memory buffer size for inputs, has also seen changes in how it’s applied and its default values. In older versions, it might have been a global setting. In newer versions, it’s often more granular per input or a global setting with a different default. If you were experiencing memory issues with large volumes of logs, you might have tuned this:
Mem_Buf_Limit 50MB
In a newer configuration, ensure this is still present or adjusted if needed, potentially as:
[SERVICE]
Mem_Buf_Limit 50MB
Or if it was specific to an input:
[INPUT]
Name tail
Path /var/log/large.log
Tag large
Mem_Buf_Limit 50MB
The exact location and scope of this directive can vary between Fluent Bit versions and plugin types, so always consult the documentation for the version you’re upgrading to.
The storage.type directive, which determines whether logs are buffered in memory or on disk, has also evolved. The filesystem type is generally preferred for durability. If you were previously using memory and experiencing data loss on restarts, migrating to filesystem is crucial. Ensure your configuration includes:
[SERVICE]
Storage.Type filesystem
Storage.Path /var/db/fluent-bit/
This ensures that logs are written to disk at /var/db/fluent-bit/ before being sent, preventing loss if Fluent Bit crashes or is restarted. The path should be a writable directory.
Finally, be aware that plugin names themselves can change or be deprecated. For instance, older versions might have had a grep filter. In newer versions, this functionality is often integrated into the grep filter with slightly different syntax or combined with other filters. Always check the plugin list for your target version. If you had a grep filter like this:
[FILTER]
Name grep
Match myapp.logs
Regex level (error|warning)
It’s likely still supported, but newer versions might offer more advanced options or alternative filters.
The next challenge you’ll likely face after a successful upgrade is optimizing output plugin configurations for specific destinations, such as Elasticsearch or Kafka, to handle increased throughput or specific data formatting requirements.