Fluent Bit’s SIGTERM handler is your last chance to save your logs when the system tells it to shut down.
Here’s Fluent Bit in action, processing logs from a file and sending them to a stdout output, with a simulated SIGTERM.
# Start Fluent Bit with a tail input and stdout output
fluent-bit -c - <<EOF
[SERVICE]
Flush 1
Daemon Off
Log_Level debug
[INPUT]
Name tail
Path /tmp/test.log
Tag my.test
[OUTPUT]
Name stdout
Match my.test
EOF
Now, create a log file:
echo "This is log line 1" > /tmp/test.log
sleep 2
echo "This is log line 2" >> /tmp/test.log
While Fluent Bit is running, send it a SIGTERM:
# Find the PID of the running fluent-bit process
PGREP_OUTPUT=$(pgrep -f "fluent-bit -c -")
echo "Fluent Bit PID: $PGREP_OUTPUT"
kill -SIGTERM $PGREP_OUTPUT
Observe the Fluent Bit output. You’ll see messages indicating it’s shutting down and attempting to flush any buffered data before exiting. If you didn’t implement a proper SIGTERM handler, logs written just before the shutdown might be lost.
The core problem Fluent Bit’s SIGTERM handler addresses is data loss during unexpected or controlled shutdowns. When Fluent Bit is running, it collects log records and often buffers them before sending them to an output destination. If the process is abruptly terminated (e.g., kill -9), any logs still in memory or in transit are lost. A SIGTERM signal, however, is a polite request to shut down. Fluent Bit’s handler intercepts this signal, enters a graceful shutdown sequence, flushes its buffers, and then exits. This ensures that as many logs as possible are delivered.
Internally, Fluent Bit manages a set of buffers for different inputs and outputs. When a SIGTERM is received, the main loop, which continuously processes events, is interrupted. The handler then triggers a flush operation across all active output plugins. This forces any pending records to be sent to their respective destinations. The Flush configuration option in the [SERVICE] section dictates how often Fluent Bit attempts to flush its buffers during normal operation, but the SIGTERM handler ensures a final, forced flush.
The primary lever you control is the Grace configuration option within the [SERVICE] section. This setting specifies the maximum time (in seconds) Fluent Bit will wait for buffered data to be flushed after receiving a SIGTERM before forcibly terminating.
[SERVICE]
Grace 5 # Wait up to 5 seconds for buffer flush
# ... other service options
If Grace is set to 0 (the default), Fluent Bit will attempt to flush immediately and exit. A non-zero value gives your output plugins more time to complete their transmissions. You can also influence this by tuning the Retry and Timeout options on your output plugins, as these affect how long individual records will be retried before being dropped during the flush process.
The Flush directive in the [SERVICE] section doesn’t directly control the SIGTERM flush duration, but it dictates the regular interval at which Fluent Bit attempts to send data. When SIGTERM is received, Fluent Bit initiates a special, final flush that bypasses this regular interval and attempts to send everything now. The Grace setting then governs how long this final, special flush is allowed to run before Fluent Bit gives up and terminates, potentially leaving some records undelivered if the output plugin is still struggling.
The next concept to grapple with is how Fluent Bit handles network interruptions during these graceful shutdowns, particularly with output plugins that rely on network connectivity.