Collect systemd Journal Logs with Fluent Bit (2026)

Fluent Bit can ingest systemd journal logs and send them to a backend, but it often fails to start up correctly if not configured to exactly match the journal’s output format.

Common Causes and Fixes

Journald Output Format Mismatch:
- Diagnosis: Check the Fluent Bit logs (journalctl -u fluent-bit -f). You’ll likely see errors like [error] [input:journald] invalid message format or [error] [parser:json] invalid json format. This means Fluent Bit is trying to parse the journal entries using a JSON parser, but journald isn’t outputting pure JSON by default.
- Fix: In your Fluent Bit configuration file (e.g., /etc/fluent-bit/fluent-bit.conf), ensure your [INPUT] section for journald specifies the Format as json and Verbosity as verbose.
```
[INPUT]
    Name              journald
    Tag               journal.docker
    Format            json
    Verbosity         verbose
    # Optional: If you only want specific units
    # Use_Docker_ID     true
    # Docker_ID_Field   container_id
```
- Why it works: Setting Format json tells Fluent Bit to expect JSON. Verbosity verbose tells journald to output all available fields in a structured, JSON-like format, which Fluent Bit’s JSON parser can then reliably consume.
Missing systemd-journal-gatewayd:
- Diagnosis: If Fluent Bit is configured for Format json and Verbosity verbose but still fails with parsing errors, it might be because the journald daemon is not running or accessible in the expected way for remote collection. While the journald input plugin can read directly from the local journal, some setups might implicitly expect a gateway service.
- Fix: Ensure the systemd-journal-gatewayd service is running and enabled.
```
sudo systemctl status systemd-journal-gatewayd
sudo systemctl enable systemd-journal-gatewayd
sudo systemctl start systemd-journal-gatewayd
```
- Why it works: Although the journald input plugin can read the local journal directly, having the gateway service running ensures a standardized interface for journal access, and sometimes Fluent Bit’s internal logic might look for it or expect it to be available for certain configurations. This is less common for direct local collection but a crucial step if remote journal access is involved or if the setup is complex.
Incorrect Tag Prefix:
- Diagnosis: You might see logs arriving at your backend (e.g., Elasticsearch, Splunk) but they are tagged incorrectly, or Fluent Bit itself throws warnings about tag conflicts. The journald input plugin often prepends journal. to the tag you define.
- Fix: If you set Tag journal.docker, the actual tag will be journal.journal.docker. Adjust your Tag in the [INPUT] section accordingly if you want a specific prefix. For example, to have the tag docker.logs, you might set:
```
[INPUT]
    Name              journald
    Tag               docker.logs # This will become journal.docker.logs
    Format            json
    Verbosity         verbose
```
  Or, more commonly, if you want the tag to be journal.docker:
```
[INPUT]
    Name              journald
    Tag               journal.docker # This will become journal.journal.docker
    Format            json
    Verbosity         verbose
```
  Correction: The default behavior is to prepend journal. to your tag. So if you want the final tag to be journal.docker, your configuration should be:
```
[INPUT]
    Name              journald
    Tag               docker # This will result in the tag "journal.docker"
    Format            json
    Verbosity         verbose
```
- Why it works: Fluent Bit uses a hierarchical tagging system. The journald input plugin has a default prefix it applies. By understanding this, you can set your desired base tag to achieve the final, prefixed tag you intend.
Permissions Issues:
- Diagnosis: Fluent Bit might fail to start with errors like [error] [input:journald] could not open journal file or [error] [io] error on open /run/log/journal/..... This indicates the user running Fluent Bit (typically fluent) doesn’t have read access to the systemd journal files.
- Fix: Add the fluent user to the systemd-journal group.
```
sudo usermod -aG systemd-journal fluent
sudo systemctl restart fluent-bit
```
- Why it works: The systemd-journal group is specifically created to grant read access to journal files to authorized users and services. Adding the Fluent Bit user to this group provides the necessary permissions.
Resource Constraints (CPU/Memory):
- Diagnosis: Fluent Bit might start but then crash or become unresponsive, showing intermittent errors like [error] [buffer] failed to flush chunk or general high CPU/memory usage in htop. Processing verbose journal output, especially from many services, can be resource-intensive.
- Fix:
  - Increase buffer sizes: In your [SERVICE] section, increase Flush and Buffer_Chunk_Size.
```
[SERVICE]
    Flush              5
    Daemon             On
    Log_Level          info
    Parsers_File       /etc/fluent-bit/parsers.conf
    Mem_Buf_Limit      50MB # Increase if needed
    Queue_Size         10000 # Increase if needed
```
  - Filter logs aggressively: In your [INPUT] or using a FILTER section, reduce the volume of logs Fluent Bit has to process.
```
[INPUT]
    Name              journald
    Tag               journal.docker
    Format            json
    Verbosity         verbose
    # Example: Only capture logs from Docker containers
    # You might need to experiment with fields available in your journal
    # Example: Filter by a specific syslog identifier or unit name
    # Match             _SYSTEMD_UNIT=docker.service
    # Match             SYSLOG_IDENTIFIER=my-app
```
  - Increase system resources: If possible, allocate more CPU or RAM to the machine running Fluent Bit.
- Why it works: Larger buffers can handle bursts of log data more gracefully, preventing memory exhaustion or write failures. Filtering reduces the processing load on Fluent Bit, allowing it to keep up with the incoming log stream.
Conflicting Input Plugins:
- Diagnosis: If you have multiple input plugins configured, especially if they are trying to read from the same source or if there’s a misconfiguration in one, it can lead to unexpected behavior or crashes. For instance, if you accidentally configure tail to read journal files or have multiple journald inputs with overlapping tags.
- Fix: Carefully review all [INPUT] sections in your Fluent Bit configuration. Ensure each input is unique and targets the intended source. If you have multiple journald inputs, make sure their Tag directives result in distinct final tags.
```
[INPUT]
    Name              journald
    Tag               docker.events
    Format            json
    Verbosity         verbose

[INPUT]
    Name              journald
    Tag               system.messages
    Format            json
    Verbosity         verbose
```
- Why it works: Each input plugin is an independent data source. By ensuring they are distinct and correctly configured, you prevent conflicts that could arise from duplicate or overlapping data ingestion paths.

The next error you’ll likely encounter after fixing these is a [error] [output:TYPE] connection refused if your output plugin (e.g., Elasticsearch, Kafka) isn’t accessible or correctly configured.