Fluent Bit can enrich log records with AWS EC2 instance metadata, which is incredibly useful for context.
Let’s see it in action.
Imagine you have logs from an application running on an EC2 instance. Without context, a log entry like {"message": "User login failed"} is just a message. But if you could automatically add the instance ID, region, AMI ID, and instance type, it becomes {"message": "User login failed", "ec2_instance_id": "i-0123456789abcdef0", "ec2_region": "us-east-1", "ec2_ami_id": "ami-0abcdef1234567890", "ec2_instance_type": "t3.medium"}. This makes debugging and analysis much more efficient, especially when dealing with a fleet of instances.
Fluent Bit achieves this using its ec2_metadata filter plugin. This plugin makes an HTTP request to the EC2 instance metadata service endpoint (http://169.254.169.254/latest/meta-data/) from within the EC2 instance itself. It fetches specific metadata fields and then adds them as new fields to your log records before they are sent to your output destination.
Here’s a simplified fluent-bit.conf snippet demonstrating this:
[SERVICE]
Flush 5
Daemon off
Log_Level info
Parsers_File parsers.conf
@INCLUDE filter_ec2_metadata.conf
@INCLUDE input_tail.conf
@INCLUDE output_stdout.conf
Now, the filter_ec2_metadata.conf file:
[FILTER]
Name ec2_metadata
Match *
Hostname ec2_instance_id
Region ec2_region
AMI_ID ec2_ami_id
Instance_Type ec2_instance_type
Security_Groups ec2_security_groups
Public_IPv4 ec2_public_ipv4
And input_tail.conf to read a sample log file:
[INPUT]
Name tail
Path /var/log/my_app.log
Tag my_app
Parser json
And output_stdout.conf to see the results:
[OUTPUT]
Name stdout
Match *
Format json
When Fluent Bit processes logs from /var/log/my_app.log (assuming it’s JSON formatted), the ec2_metadata filter will intercept each record. For each record matching * (all records), it will query the metadata service. It will then add fields like ec2_instance_id, ec2_region, etc., to the record. The stdout output will then print these enriched JSON logs.
The key configuration parameters for the ec2_metadata filter are straightforward:
Hostname: The key name for the instance ID.Region: The key name for the AWS region.AMI_ID: The key name for the AMI ID.Instance_Type: The key name for the instance type.Security_Groups: The key name for the associated security groups.Public_IPv4: The key name for the public IPv4 address.
You can specify any key names you prefer for these metadata fields. The plugin defaults to using the names shown above if not specified.
The magic happens because Fluent Bit runs on the EC2 instance. The metadata service is a special, non-routable IP address (169.254.169.254) accessible only from within the instance. This means the plugin doesn’t need explicit AWS credentials to fetch this information; it’s implicitly available to the running instance.
The ec2_metadata filter plugin works by periodically making requests to the EC2 instance metadata service. It caches the retrieved metadata for a certain period (defaulting to 60 seconds) to avoid overwhelming the metadata service with requests for every single log record. This caching mechanism is crucial for performance and to stay within the metadata service’s rate limits. If you need more granular control over the cache duration, you can use the Metadata_Cache_TTL option within the [FILTER] block, setting it to a desired number of seconds.
A common misconception is that you need to configure IAM roles or access keys for this to work. That’s not the case for fetching instance metadata. The metadata service is designed for this exact purpose, providing information about the instance itself without requiring explicit AWS credentials. However, if you were trying to access other AWS services (like S3 or CloudWatch Logs), you would indeed need appropriate IAM permissions.
The next step after enriching logs with instance metadata is often to use this enriched data for more intelligent routing or filtering.