Articles

Watch Log Files on Linux with Fluent Bit Inotify (2026)

April 18, 2026 5 min read

Watching log files on Linux with Fluent Bit’s inotify input is more about understanding how file system events trigger processing than about the inotify mechanism itself.

Here’s Fluent Bit watching /var/log/myapp.log for changes:

[SERVICE]
    Flush        5
    Daemon       on
    Log_Level    info
    Parsers_File parsers.conf
    HTTP_Server  on
    HTTP_Listen  127.0.0.1
    HTTP_Port    2020

[INPUT]
    Name        tail
    Tag         myapp.log
    Path        /var/log/myapp.log
    Parser      docker
    DB          /var/log/flb_myapp.db
    Mem_Buffer_Limit 1MB

When /var/log/myapp.log is appended to, Fluent Bit’s tail input, configured with inotify enabled (which is the default behavior for tail when inotify_watch_enabled is not explicitly set to false), receives a notification. This notification tells Fluent Bit that the file has been modified. Fluent Bit then reads the new content from the last known position, parses it according to the docker parser, and tags it as myapp.log. The DB file /var/log/flb_myapp.db stores the last read position, ensuring that even if Fluent Bit restarts, it resumes reading from where it left off.

The core problem Fluent Bit’s tail input with inotify solves is efficient, low-overhead log monitoring. Instead of constantly polling files for changes (which is CPU-intensive), inotify leverages the Linux kernel’s file system event notification system. This means the kernel actively tells Fluent Bit when a file it’s watching has been written to. This is crucial for performance, especially when monitoring many large log files.

Let’s break down the tail input configuration:

  • Name tail: This specifies the input plugin. tail is designed for reading log files.
  • Tag myapp.log: This assigns a tag to all records read from this input. Tags are used for routing and filtering data within Fluent Bit.
  • Path /var/log/myapp.log: The absolute path to the log file to be monitored.
  • Parser docker: This tells Fluent Bit to use the docker parser to interpret the log lines. Parsers are essential for structuring unstructured log data. The docker parser is commonly used for logs generated by Docker containers.
  • DB /var/log/flb_myapp.db: This is a state database. It stores the last read position of each file being tailed. This allows Fluent Bit to resume tailing without missing any log entries if it’s stopped and restarted.
  • Mem_Buffer_Limit 1MB: This sets a limit on the amount of data that can be buffered in memory before being flushed.

Consider a scenario where you have multiple applications generating logs in /var/log/apps/. You can configure Fluent Bit to watch all of them:

[INPUT]
    Name        tail
    Tag         apps.*
    Path        /var/log/apps/*.log
    Parser      json
    DB          /var/log/flb_apps.db

Here, Path /var/log/apps/*.log uses a wildcard to match any file ending in .log within the /var/log/apps/ directory. The Tag apps.* uses a wildcard in the tag, meaning any log file matched by the path will have a tag starting with apps.. For example, a log from /var/log/apps/user_service.log might get the tag apps.user_service.log. The Parser json indicates that these logs are expected to be in JSON format.

The most surprising aspect of inotify in this context is how seamlessly it integrates with file system operations. It’s not a separate daemon or process that Fluent Bit has to manage; it’s a direct kernel mechanism. When Fluent Bit’s tail input is configured, it registers watches with the kernel for the specified files. The kernel then issues IN_MODIFY events (among others) to Fluent Bit whenever data is appended to these files. Fluent Bit’s internal event loop picks these up and acts upon them, reading the new data. This event-driven model is why it’s so efficient.

The tail input plugin, by default, enables inotify monitoring. You can explicitly disable it with inotify_watch_enabled false, but this would revert to polling, which is generally less efficient. The inotify mechanism is what allows Fluent Bit to react to new log entries almost instantaneously without consuming significant CPU resources. The DB file is critical for durability; without it, Fluent Bit would re-read logs from the beginning after every restart, leading to duplicate entries and potential data loss if logs are rotated.

When dealing with log rotation (e.g., myapp.log being renamed to myapp.log.1 and a new myapp.log being created), the tail input handles this gracefully. By default, tail monitors file descriptors rather than just file paths. When a file is rotated, the old file descriptor points to the renamed file, and a new file descriptor is opened for the new myapp.log. Fluent Bit will continue to tail the original file descriptor (which is now myapp.log.1) and will also detect the new myapp.log via inotify and start tailing it. This is why the DB file is essential; it helps Fluent Bit track progress across these file descriptor changes and renames.

If you need to monitor directories for new files being created, not just appends to existing files, you would use the inotify input plugin directly, not the tail input. The tail input is specifically for monitoring existing files for changes. The inotify input plugin allows you to watch directories for events like IN_CREATE, IN_DELETE, IN_MOVED_TO, etc., and trigger actions based on those events, often used for processing new files as they appear.

Understanding the distinction between the tail input (which uses inotify implicitly for efficiency) and the inotify input plugin (which is explicitly for reacting to file system events on directories and files) is key to leveraging Fluent Bit for robust log collection. The tail input’s strength lies in its ability to efficiently tail existing files, while the inotify input plugin offers broader file system event monitoring.

The next step after mastering file monitoring is often setting up output plugins to send this collected data to your chosen destination.

Want structured learning?

Take the full Fluentbit course →