Fluentd’s latest version has a fundamentally different approach to plugin management that makes upgrading a snap, provided you’re willing to adopt a more declarative style.
Let’s see it in action. Imagine you have a basic Fluentd configuration sending logs to stdout:
<source>
@type forward
port 24224
</source>
<match **>
@type stdout
</match>
To upgrade this to use the latest features, you’d typically start by updating your fluentd gem:
gem update fluentd
However, the real power comes from how plugins are now handled. Instead of relying on gem install and then referencing them directly in the config, you use the fluentd-plugin command to manage them. If you wanted to add a buffer plugin to your stdout output, you’d first install it:
fluentd-plugin install fluent-plugin-buffer
And then your configuration would look something like this:
<source>
@type forward
port 24224
</source>
<match **>
@type stdout
<buffer>
@type file
path /var/log/td-agent/buffer/stdout
</buffer>
</match>
This shift from imperative (gem install) to declarative (<buffer>) configuration is the core of the upgrade. It means Fluentd itself now manages plugin dependencies and loading more intelligently.
The problem this solves is the notorious "dependency hell" that plagued older Fluentd versions. Installing plugins often led to version conflicts between different plugins, requiring manual intervention or complex gem version pinning. The new model centralizes plugin management through the fluentd-plugin CLI, ensuring that compatible versions are installed and loaded.
Internally, the fluentd-plugin command interacts with Fluentd’s plugin registry. When you run fluentd-plugin install <plugin_name>, it checks for available versions, resolves dependencies against already installed plugins, and then installs the necessary gems. Fluentd, upon startup, consults this registry to find and load the required plugins based on their @type directives in the configuration. This makes the entire process more robust and predictable.
The exact levers you control are primarily the @type parameters within your configuration and the commands you use with fluentd-plugin. For instance, if you’re using a custom plugin that was previously installed via gem install my-custom-plugin, you’d now manage it with fluentd-plugin install my-custom-plugin. The configuration would then simply reference its @type as defined by the plugin author.
When you upgrade, you’ll likely encounter a new error related to plugin loading if you haven’t fully transitioned. The system might complain about an unrecognized @type for a plugin you previously installed via gem. This is because Fluentd’s loader now relies on the fluentd-plugin management system.
The most surprising thing about this upgrade path is how it decouples plugin installation from Fluentd’s core lifecycle. You can now install, update, or remove plugins independently of restarting Fluentd itself, as long as you’re using the fluentd-plugin CLI. This allows for more granular control and testing of individual plugin changes without impacting the entire logging pipeline.
The next concept you’ll likely run into is the structured logging support, which offers a more powerful way to filter and route events.