Fluentd’s relabel and route directives let you dynamically redirect and transform log streams without duplicating configuration, making massive Fluentd deployments manageable.
Let’s see this in action. Imagine we have a basic Fluentd setup receiving logs from a web server and an application server.
# Base configuration for web server logs
<source>
@type tail
path /var/log/nginx/access.log
pos_file /var/log/td-agent/pos/nginx-access.pos
tag web.access
<parse>
@type nginx
</parse>
</source>
# Base configuration for application server logs
<source>
@type tail
path /var/log/my_app/application.log
pos_file /var/log/td-agent/pos/my_app-application.pos
tag app.application
<parse>
@type json
</parse>
</source>
# Default output for all logs (initially)
<match **>
@type stdout
</match>
This setup works, but as we add more services, the <match> blocks will become a sprawling mess. We want to send web server logs to one Elasticsearch cluster and application logs to another, potentially with different processing for each.
This is where relabel and route shine. relabel allows us to change the tag of a log record based on its current tag, and route lets us define where those relabeled records should go.
Here’s how we can refactor the above to use relabel and route for more sophisticated routing:
# --- Sources ---
<source>
@type tail
path /var/log/nginx/access.log
pos_file /var/log/td-agent/pos/nginx-access.pos
tag web.access
<parse>
@type nginx
</parse>
</source>
<source>
@type tail
path /var/log/my_app/application.log
pos_file /var/log/td-agent/pos/my_app-application.pos
tag app.application
<parse>
@type json
</parse>
</source>
# --- Relabeling and Routing ---
# Relabel web access logs to a specific tag for routing
<filter **>
@type relabel
@label web_logs
# Matches any tag that starts with 'web.'
# If it matches, it will be sent to the 'web_logs' label
<rule>
key tag
pattern /^web\./
action label
</rule>
</filter>
# Relabel application logs to a specific tag for routing
<filter **>
@type relabel
@label app_logs
# Matches any tag that starts with 'app.'
# If it matches, it will be sent to the 'app_logs' label
<rule>
key tag
pattern /^app\./
action label
</rule>
</filter>
# --- Output Routes ---
# Route for web logs
<route>
@id web_route
@label web_logs
# Now, send the relabeled 'web_logs' to its destination
<match web.**>
@type elasticsearch
host elasticsearch-web.example.com
port 9200
logstash_format true
logstash_prefix web-logs
include_tag_key true
tag_key @log_name
</match>
</route>
# Route for application logs
<route>
@id app_route
@label app_logs
# Now, send the relabeled 'app_logs' to its destination
<match app.**>
@type elasticsearch
host elasticsearch-app.example.com
port 9200
logstash_format true
logstash_prefix app-logs
include_tag_key true
tag_key @log_name
</match>
</route>
# Fallback route for any logs not caught by the above
# This is crucial to avoid losing logs if a rule isn't matched.
<route>
@id fallback_route
# Matches anything not explicitly routed by @label
# You could send this to a general logging system or stdout for debugging
<match **>
@type stdout
</match>
</route>
In this refactored setup, the <source> sections remain the same, defining where logs come from and their initial tags (web.access, app.application).
The <filter> sections with @type relabel are the core of the dynamic routing.
- The first
relabelfilter looks for any incoming record whosetagmatches the pattern/^web\./. If it matches, it applies thelabelaction, effectively sending this record to a new "virtual stream" or "label" namedweb_logs. - The second
relabelfilter does the same for tags starting withapp., directing them to theapp_logslabel.
The <route> blocks then pick up these labeled streams.
- The
web_routeis configured with@label web_logs. This means it will only process records that have been directed to theweb_logslabel by a precedingrelabelfilter. Inside this route, a<match web.**>directive handles the actual output to the web Elasticsearch cluster. Theweb.**pattern here is a bit redundant given therelabel’s action, but it’s good practice to have a specific match within the route. - Similarly,
app_routelistens for theapp_logslabel and outputs to the application Elasticsearch cluster.
Finally, the fallback_route ensures that any logs not caught by the specific relabel and route configurations are still processed, preventing data loss.
The real power here is that you can add many more sources, and as long as their tags follow a consistent pattern (e.g., api.access, db.query, cache.hit), you can create new relabel filters and corresponding route blocks to send them to different destinations without touching the existing output configurations. The relabel directive essentially acts as a traffic cop, directing log streams to different named lanes (@label) that the route directives then pick up and send to their final destinations.
The most surprising thing about relabel and route is that they operate independently of the final output plugins. A relabel filter doesn’t know where the logs are going, only that they should be tagged for a specific route. A route directive doesn’t know how the logs arrived at its label, only that it needs to process records associated with that label. This separation of concerns is what allows for incredibly flexible and scalable log routing architectures.
When you have a complex setup with many sources and destinations, and you need to apply different processing (e.g., adding metadata, filtering sensitive information) before sending logs to their final destinations, you can insert additional <filter> blocks within the <route> that perform these transformations. For example, within the web_route, you could add a <filter> block before the <match> to enrich web logs with GeoIP data before sending them to Elasticsearch.
The next step in managing complex Fluentd configurations is to abstract common filtering and output patterns into reusable configuration files using Fluentd’s <include> directive, allowing you to define standard processing pipelines for different log types.