Fluentd’s relabel and route directives let you dynamically redirect and transform log streams without duplicating configuration, making massive Fluentd deployments manageable.

Let’s see this in action. Imagine we have a basic Fluentd setup receiving logs from a web server and an application server.

# Base configuration for web server logs
<source>
  @type tail
  path /var/log/nginx/access.log
  pos_file /var/log/td-agent/pos/nginx-access.pos
  tag web.access
  <parse>
    @type nginx
  </parse>
</source>

# Base configuration for application server logs
<source>
  @type tail
  path /var/log/my_app/application.log
  pos_file /var/log/td-agent/pos/my_app-application.pos
  tag app.application
  <parse>
    @type json
  </parse>
</source>

# Default output for all logs (initially)
<match **>
  @type stdout
</match>

This setup works, but as we add more services, the <match> blocks will become a sprawling mess. We want to send web server logs to one Elasticsearch cluster and application logs to another, potentially with different processing for each.

This is where relabel and route shine. relabel allows us to change the tag of a log record based on its current tag, and route lets us define where those relabeled records should go.

Here’s how we can refactor the above to use relabel and route for more sophisticated routing:

# --- Sources ---
<source>
  @type tail
  path /var/log/nginx/access.log
  pos_file /var/log/td-agent/pos/nginx-access.pos
  tag web.access
  <parse>
    @type nginx
  </parse>
</source>

<source>
  @type tail
  path /var/log/my_app/application.log
  pos_file /var/log/td-agent/pos/my_app-application.pos
  tag app.application
  <parse>
    @type json
  </parse>
</source>

# --- Relabeling and Routing ---

# Relabel web access logs to a specific tag for routing
<filter **>
  @type relabel
  @label web_logs
  # Matches any tag that starts with 'web.'
  # If it matches, it will be sent to the 'web_logs' label
  <rule>
    key tag
    pattern /^web\./
    action label
  </rule>
</filter>

# Relabel application logs to a specific tag for routing
<filter **>
  @type relabel
  @label app_logs
  # Matches any tag that starts with 'app.'
  # If it matches, it will be sent to the 'app_logs' label
  <rule>
    key tag
    pattern /^app\./
    action label
  </rule>
</filter>

# --- Output Routes ---

# Route for web logs
<route>
  @id web_route
  @label web_logs
  # Now, send the relabeled 'web_logs' to its destination
  <match web.**>
    @type elasticsearch
    host elasticsearch-web.example.com
    port 9200
    logstash_format true
    logstash_prefix web-logs
    include_tag_key true
    tag_key @log_name
  </match>
</route>

# Route for application logs
<route>
  @id app_route
  @label app_logs
  # Now, send the relabeled 'app_logs' to its destination
  <match app.**>
    @type elasticsearch
    host elasticsearch-app.example.com
    port 9200
    logstash_format true
    logstash_prefix app-logs
    include_tag_key true
    tag_key @log_name
  </match>
</route>

# Fallback route for any logs not caught by the above
# This is crucial to avoid losing logs if a rule isn't matched.
<route>
  @id fallback_route
  # Matches anything not explicitly routed by @label
  # You could send this to a general logging system or stdout for debugging
  <match **>
    @type stdout
  </match>
</route>

In this refactored setup, the <source> sections remain the same, defining where logs come from and their initial tags (web.access, app.application).

The <filter> sections with @type relabel are the core of the dynamic routing.

  • The first relabel filter looks for any incoming record whose tag matches the pattern /^web\./. If it matches, it applies the label action, effectively sending this record to a new "virtual stream" or "label" named web_logs.
  • The second relabel filter does the same for tags starting with app., directing them to the app_logs label.

The <route> blocks then pick up these labeled streams.

  • The web_route is configured with @label web_logs. This means it will only process records that have been directed to the web_logs label by a preceding relabel filter. Inside this route, a <match web.**> directive handles the actual output to the web Elasticsearch cluster. The web.** pattern here is a bit redundant given the relabel’s action, but it’s good practice to have a specific match within the route.
  • Similarly, app_route listens for the app_logs label and outputs to the application Elasticsearch cluster.

Finally, the fallback_route ensures that any logs not caught by the specific relabel and route configurations are still processed, preventing data loss.

The real power here is that you can add many more sources, and as long as their tags follow a consistent pattern (e.g., api.access, db.query, cache.hit), you can create new relabel filters and corresponding route blocks to send them to different destinations without touching the existing output configurations. The relabel directive essentially acts as a traffic cop, directing log streams to different named lanes (@label) that the route directives then pick up and send to their final destinations.

The most surprising thing about relabel and route is that they operate independently of the final output plugins. A relabel filter doesn’t know where the logs are going, only that they should be tagged for a specific route. A route directive doesn’t know how the logs arrived at its label, only that it needs to process records associated with that label. This separation of concerns is what allows for incredibly flexible and scalable log routing architectures.

When you have a complex setup with many sources and destinations, and you need to apply different processing (e.g., adding metadata, filtering sensitive information) before sending logs to their final destinations, you can insert additional <filter> blocks within the <route> that perform these transformations. For example, within the web_route, you could add a <filter> block before the <match> to enrich web logs with GeoIP data before sending them to Elasticsearch.

The next step in managing complex Fluentd configurations is to abstract common filtering and output patterns into reusable configuration files using Fluentd’s <include> directive, allowing you to define standard processing pipelines for different log types.

Want structured learning?

Take the full Fluentd course →