Storing sensitive information like API keys or passwords directly in configuration files is a recipe for disaster. Environment variables offer a more secure and flexible way to manage these credentials for Fluentd.
Let’s see this in action. Imagine a Fluentd configuration that needs to send logs to a third-party service requiring an API key. Instead of hardcoding it, we’ll use an environment variable.
<match *.log>
@type forward
<server>
host logs.example.com
port 24224
</server>
<buffer>
flush_interval 5s
</buffer>
<format>
@type json
</format>
<storage>
@type file
path /var/log/td-agent/buffer/my_app
</storage>
# Using an environment variable for authentication
<http_header>
X-API-KEY ${ENV_THIRD_PARTY_API_KEY}
</http_header>
</match>
Here, ${ENV_THIRD_PARTY_API_KEY} is a placeholder that Fluentd will resolve to the value of the THIRD_PARTY_API_KEY environment variable when it starts up.
This approach solves the problem of credential exposure. When Fluentd starts, it reads its configuration. If it encounters a placeholder like ${ENV_VAR_NAME}, it looks for an environment variable with that name in the operating system’s environment. If found, it substitutes the variable’s value into the configuration. This means the sensitive API key never appears directly in the Fluentd configuration file itself.
The core mechanism is Fluentd’s built-in support for environment variable substitution, often referred to as "variable interpolation." This feature is designed to inject dynamic values into your configuration at runtime. You can access environment variables using the ${ENV_VAR_NAME} syntax. This applies to any configuration parameter, not just HTTP headers. For instance, you could use it for database connection strings, API endpoints, or any other sensitive or dynamic value.
To make this work, you need to set the environment variable before starting the Fluentd process. How you do this depends on your operating system and how you’re running Fluentd.
For a typical Linux system using systemd, you’d edit the Fluentd service unit file. For example, if your service is named td-agent.service, you might edit /etc/systemd/system/td-agent.service.d/override.conf (or create it if it doesn’t exist) and add:
[Service]
Environment="THIRD_PARTY_API_KEY=your_super_secret_api_key_here"
Environment="ANOTHER_SECRET=another_value"
Then, reload the systemd daemon and restart Fluentd:
sudo systemctl daemon-reload
sudo systemctl restart td-agent
If you’re running Fluentd manually from the command line, you’d export the variable first:
export THIRD_PARTY_API_KEY="your_super_secret_api_key_here"
fluentd -c /etc/fluent/fluent.conf
Or, on systems that support it, you can prepend it directly:
THIRD_PARTY_API_KEY="your_super_secret_api_key_here" fluentd -c /etc/fluent/fluent.conf
The primary benefit is security. Credentials are not committed to version control, and they aren’t visible in plain text in your Fluentd configuration files, which might be readable by more users than necessary. It also enhances flexibility; you can deploy the same Fluentd configuration across different environments (development, staging, production) by simply setting different environment variables without altering the configuration file itself.
A common pitfall is forgetting to restart the Fluentd service after setting environment variables, especially when using systemd. The Environment= directive in a systemd service file only takes effect when the service is started or restarted. Simply editing the configuration file won’t make Fluentd pick up the new variable; it needs to re-read its environment.
The next step is often to use more advanced templating or secrets management systems for even greater control and security.