Fluent Bit’s trace mode is surprisingly powerful for understanding how data flows, but most people only use it for basic debugging.
Let’s see what happens when we send some logs and inspect them with trace mode.
First, we’ll set up a simple Fluent Bit configuration to read from standard input and write to standard output.
[SERVICE]
Log_Level debug
Parsers_File parsers.conf
[INPUT]
Name tail
Path /dev/stdin
Tag stdin.log
Format json
[OUTPUT]
Name stdout
Match *
Format json
Now, let’s create a parsers.conf file with a basic JSON parser:
[PARSER]
Name json
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%LZ
With this setup, if we pipe some JSON data into Fluent Bit, we can see it processed.
echo '{"message": "hello world", "time": "2023-10-27T10:00:00.000Z"}' | fluent-bit -c fluent-bit.conf
The output will be a JSON object, but Fluent Bit adds some metadata. Notice the tag and time fields it generates.
{
"time": "2023-10-27T10:00:00.123Z",
"tag": "stdin.log",
"message": "hello world"
}
The real magic happens when we enable Trace=On in the [SERVICE] section. This drastically increases the verbosity of the logs, showing every single event as it’s processed by different plugins.
[SERVICE]
Log_Level debug
Parsers_File parsers.conf
Trace On
[INPUT]
Name tail
Path /dev/stdin
Tag stdin.log
Format json
[OUTPUT]
Name stdout
Match *
Format json
Running the same command now will produce a torrent of output. You’ll see lines prefixed with [trace], detailing which plugin is handling an incoming record, what transformations are being applied, and what data is being passed along. This is invaluable for pinpointing where a record might be dropped or malformed.
Here’s a snippet of what you might see with Trace=On:
[0] stdin.log: [trace] plugin=tail, action=read, record={"message": "hello world", "time": "2023-10-27T10:00:00.000Z"}
[0] stdin.log: [trace] plugin=parser, action=parse, record={"message": "hello world", "time": "2023-10-27T10:00:00.000Z"}
[0] stdin.log: [trace] plugin=parser, action=parse_success, record={"message": "hello world", "time": "2023-10-27T10:00:00.000Z", "tag": "stdin.log"}
[0] stdin.log: [trace] plugin=stdout, action=write, record={"message": "hello world", "time": "2023-10-27T10:00:00.000Z", "tag": "stdin.log"}
The key levers you control are the Log_Level and Trace directives in the [SERVICE] section. Log_Level can be set to error, warning, info, debug, or trace. Trace=On essentially forces the Log_Level to trace and enables detailed event-by-event logging. You can also filter trace output by using the Trace_Key and Trace_Value options, allowing you to focus on specific records or tags.
The Parsers_File directive is crucial for defining how Fluent Bit interprets incoming data. Without a correctly defined parser, especially for structured formats like JSON or the Elastic Common Schema (ECS), your logs might not be parsed as expected, leading to unexpected behavior downstream. Fluent Bit uses a robust parsing engine that supports regular expressions, JSON, Lua, and more.
One aspect of Fluent Bit’s processing that catches many users off guard is the concept of "tags" and how they are dynamically modified. When data enters Fluent Bit, it’s assigned a tag (e.g., stdin.log). This tag is then used for routing data to specific output plugins via the Match directive. However, tags can be altered by plugins, especially by parsers or filters. For instance, a parser might extract a field and use it to construct a new tag, effectively changing the record’s identity for routing purposes. Understanding this tag manipulation is vital for ensuring data reaches its intended destination.
When debugging complex pipelines involving multiple filters, the order of operations becomes paramount. Filters are applied sequentially to records, and the output of one filter becomes the input for the next. If a filter modifies a record in a way that’s incompatible with a subsequent filter, or if a filter incorrectly drops records, tracing the flow with Trace=On will reveal exactly where the data deviates from your expectations.
The next logical step in mastering Fluent Bit is understanding how to write custom Lua filters to perform complex data transformations that go beyond standard plugin capabilities.