Fluentd can fan out log events to multiple destinations at once, not by duplicating the event itself, but by creating multiple independent "paths" for the data.
Here’s Fluentd processing logs and sending them to both a file and Elasticsearch:
<source>
@type tail
path /var/log/my_app.log
pos_file /var/log/td-agent/my_app.log.pos
tag myapp.log
<parse>
@type json
</parse>
</source>
<match myapp.log>
@type copy
<store>
@type file
path /var/log/fluentd/myapp.log.%Y%m%d
buffer_type file
buffer_path /var/log/td-agent/buffer/myapp
flush_interval 5s
</store>
<store>
@type elasticsearch
host localhost
port 9200
index_name myapp-%Y%m%d
type_name myapp_log
flush_interval 5s
</store>
</match>
When Fluentd receives an event tagged myapp.log, the <match myapp.log> directive captures it. Instead of directly sending it to a single destination, the @type copy plugin acts as a splitter. Each <store> block within the copy plugin defines an independent output destination. The copy plugin then sends a copy of the incoming event to each of these <store> blocks.
The core problem this solves is avoiding a single point of failure and enabling diverse analysis. If you only sent logs to Elasticsearch, and Elasticsearch was down, you’d lose logs. By copying to a file, you have a local backup. Similarly, you might want to aggregate logs in a central system like Elasticsearch for real-time analysis, while also archiving them to S3 for long-term storage. The copy plugin makes this trivially achievable without complex routing logic or event duplication at the source.
Internally, the copy plugin doesn’t modify the event itself. It simply takes the received event and passes it along to each configured <store> plugin. Each <store> plugin then handles the event independently, applying its own buffering, retries, and formatting. This means if the Elasticsearch output fails to send an event, the file output can still succeed, and vice-versa.
The key levers you control are the <store> blocks. Each one is a fully independent Fluentd output plugin. You can mix and match any output plugin that Fluentd supports: file, Elasticsearch, S3, Kafka, Splunk, etc. You can also apply different buffering strategies, flush_interval settings, and formatting rules to each output independently within their respective <store> blocks. This allows for highly customized data flow based on the needs of each destination.
The buffer_type file combined with buffer_path in the file output is crucial for durability. If Fluentd itself crashes after reading a log but before successfully writing it to the file, the buffer file on disk ensures that the event isn’t lost. When Fluentd restarts, it will attempt to flush these buffered events again.
What most people overlook is how the copy plugin interacts with tags. If you have multiple <match> directives for the same tag, they will all trigger. However, if you use the copy plugin, it’s a single <match> directive that then fans out. This distinction is important for performance and clarity, as it consolidates the matching logic.
The next challenge you’ll face is handling different data formats or transformations for each output, which often leads to using the filter plugin within individual <store> blocks.