Fluent Bit can send records to multiple destinations based on pattern matching of their tags.
Let’s see it in action. Imagine you have logs from different services, and you want to send application logs to Elasticsearch, system logs to Splunk, and anything else to a local file.
Here’s a simplified fluent-bit.conf that accomplishes this:
[SERVICE]
Flush 5
Daemon off
Log_Level info
[INPUT]
Name tail
Path /var/log/app/*.log
Tag app.*
[INPUT]
Name tail
Path /var/log/syslog
Tag syslog.*
[INPUT]
Name tail
Path /var/log/other/*.log
Tag other.*
[OUTPUT]
Name es
Match app.*
Host elasticsearch.example.com
Port 9200
Logstash_Format on
Replace_Dots on
[OUTPUT]
Name splunk
Match syslog.*
Host splunk.example.com
Port 8088
Splunk_Token YOUR_SPLUNK_TOKEN
Splunk_Source app_logs
Splunk_Sourcetype syslog
[OUTPUT]
Name file
Match other.*
Path /var/log/fluent-bit/fallback.log
In this configuration:
- The
[SERVICE]section sets up basic Fluent Bit behavior. - We have three
[INPUT]sections. Thetailinput reads from specified files.- The first
tailinput reads all files ending in.logwithin/var/log/app/and assigns them the tagapp.*. - The second reads
/var/log/syslogand tags itsyslog.*. - The third reads files in
/var/log/other/and tags themother.*.
- The first
- Then, we have three
[OUTPUT]sections. The crucial part is theMatchdirective.- The
esoutput usesMatch app.*. This means any record whose tag starts withapp.will be sent to Elasticsearch. - The
splunkoutput usesMatch syslog.*. Records tagged withsyslog.*will go to Splunk. - The
fileoutput usesMatch other.*. Anything tagged withother.*will be written to/var/log/fluent-bit/fallback.log.
- The
The Match directive supports glob-style wildcard matching. The asterisk (*) matches zero or more characters. So, app.* will match tags like app.service1, app.users, or even just app. This allows for flexible routing based on how you structure your tags. You can also use ? for a single character match, or [...] for character set matching.
The most surprising true thing about this is that the Match directive doesn’t just filter which output gets the record; it also influences how the record is processed by that output plugin. Some plugins might have specific configurations that are only enabled or altered when a particular tag pattern is matched. For example, an output might send a record to a different index in Elasticsearch if its tag matches app.production.* versus app.staging.*. This allows for granular control over data routing and storage based on the origin and context conveyed by the tag.
The next concept you’ll want to explore is how to combine multiple Match patterns within a single output.