Installing and managing Fluentd plugins via RubyGems is the standard way to extend Fluentd’s functionality, but it’s surprisingly easy to get tangled up in dependency conflicts.
Let’s see Fluentd in action with a common setup: ingesting Nginx logs and sending them to Elasticsearch.
First, we need a basic Fluentd configuration (fluent.conf).
<source>
@type tail
path /var/log/nginx/access.log
pos_file /var/log/td-agent/td-agent.pos
tag nginx.access
<parse>
@type nginx
</parse>
</source>
<match nginx.access>
@type elasticsearch
host localhost
port 9200
logstash_format true
logstash_prefix nginx-access
include_tag_key true
tag_key logstash
flush_interval 5s
</match>
Now, imagine we want to add a filter to enrich our logs with GeoIP information before sending them to Elasticsearch. We’ll need the fluent-plugin-geoip plugin.
To install it, we’d typically run:
/usr/sbin/td-agent-gem install fluent-plugin-geoip
This command uses the gem executable bundled with td-agent (Fluentd’s packaged version) to install the plugin. It downloads the fluent-plugin-geoip gem and its dependencies, then places them in td-agent’s gem directory.
After installation, we modify our fluent.conf to include the filter:
<source>
@type tail
path /var/log/nginx/access.log
pos_file /var/log/td-agent/td-agent.pos
tag nginx.access
<parse>
@type nginx
</parse>
</source>
<filter nginx.access>
@type geoip
key_name remote_addr
backend_database geoip/GeoLite2-City.mmdb
backend_result_key geoip
</filter>
<match nginx.access>
@type elasticsearch
host localhost
port 9200
logstash_format true
logstash_prefix nginx-access
include_tag_key true
tag_key logstash
flush_interval 5s
</match>
And then restart td-agent:
sudo systemctl restart td-agent
The GeoIP plugin will now look up the IP address from each Nginx log record and add geographical information to the event.
The core problem Fluentd plugins solve is modularity. Instead of a monolithic logging agent, you have a core engine that can load specialized components (sources, filters, outputs) as needed. This allows for a vast ecosystem of plugins for almost any data source, processing step, or destination.
Internally, when Fluentd starts, it scans its configured gem paths for installed plugins. When it encounters a directive like <source>, <filter>, or <match>, it looks for a corresponding plugin class (e.g., Fluent::Plugin::TailInput, Fluent::Plugin::GeoipFilter, Fluent::Plugin::ElasticsearchOutput) and instantiates it. The configuration within the directive is then passed to the plugin’s initialization method.
The td-agent-gem command is crucial because it installs gems into the specific directory that td-agent is configured to look for plugins. If you use the system gem install command, td-agent might not find your plugins unless you manually configure its gem path.
When you install a plugin like fluent-plugin-geoip, it often has its own dependencies. For instance, fluent-plugin-geoip might depend on geoip and hiredis gems. RubyGems handles this dependency resolution. However, if two plugins you install require different, incompatible versions of the same dependency, you’ll run into trouble. This is where td-agent-gem can sometimes be more forgiving as it manages gems specifically for the td-agent environment.
The backend_database parameter in the GeoIP filter is particularly interesting. It points to a MaxMind GeoLite2 City database file. Fluentd doesn’t ship with this file; you’d need to download it separately from MaxMind (often requiring registration) and place it in a location accessible by Fluentd. This highlights that some plugins require external data or services to function correctly.
Managing plugin versions is key. If a plugin update breaks compatibility, you might need to downgrade. You can check installed plugins and their versions with:
/usr/sbin/td-agent-gem list
And install a specific version with:
/usr/sbin/td-agent-gem install fluent-plugin-geoip --version 1.0.2
If you encounter a LoadError or NameError when starting Fluentd, it almost always means a plugin isn’t installed correctly or its dependencies are missing. The error message will usually point to the missing class or file.
The next step after successfully installing and configuring plugins is often optimizing their performance, which involves understanding buffering mechanisms and concurrency settings.