Loki systemd Logs: Scrape journald with Promtail (2026)

Promtail is failing to scrape logs from systemd journald, resulting in missing logs in Loki.

The core issue is that Promtail, when configured to scrape journald, is unable to establish a connection or properly read from the journald socket, preventing it from ingesting logs into Loki. This isn’t a simple permissions issue, but rather a complex interplay of systemd’s internal workings, Promtail’s client implementation, and the underlying Linux capabilities.

Here are the most common reasons this fails and how to fix them:

1. Incorrect `journald` Path in Promtail Configuration

Promtail needs to know where to find the journald socket. Often, people assume a default path without verifying.

Diagnosis: Check your Promtail configuration file (usually promtail-config.yaml). Look for the journal scrape configuration block.

Common Mistake:

scrape_configs:
  - job_name: journal
    journal:
      path: /var/log/journal/ # This is often incorrect
    relabel_configs:
      # ...

The Fix: The path directive for journal scraping in Promtail is not the directory containing journal files. It should point to the journald socket file. The standard path is /run/systemd/journal/stdout.
```
scrape_configs:
  - job_name: journal
    journal:
      path: /run/systemd/journal/stdout # Correct path for journald socket
    relabel_configs:
      # ...
```
Why it Works: Promtail uses this path to establish a direct connection to the systemd journald daemon, allowing it to stream logs in real-time rather than trying to parse static log files.

2. Missing `CAP_DAC_READ_SEARCH` Capability for Promtail

Journald logs are often protected, and Promtail needs specific capabilities to access them. The CAP_DAC_READ_SEARCH capability allows a process to bypass file read and execute permission checks.

Diagnosis: If Promtail is running as a non-root user (which is good practice), it might not have the necessary privileges. You can check the capabilities of a running process with getpcaps <pid>.
The Fix (using systemd service): If you’re running Promtail as a systemd service, you can grant it the required capability. Edit your promtail.service file (e.g., /etc/systemd/system/promtail.service or a drop-in file in /etc/systemd/system/promtail.service.d/) and add:
```
[Service]
CapabilityBoundingSet=CAP_DAC_READ_SEARCH CAP_CHOWN
AmbientCapabilities=CAP_DAC_READ_SEARCH CAP_CHOWN
```
Then reload systemd and restart Promtail:
```
sudo systemctl daemon-reload
sudo systemctl restart promtail
```
Why it Works: By explicitly granting CAP_DAC_READ_SEARCH (and often CAP_CHOWN if Promtail needs to manage its data directory permissions), you allow the Promtail process to read from the journald socket without needing to run as the root user, which is a security risk.

3. Journald Not Running or Accessible

This is fundamental. If systemd-journald itself isn’t running or configured to output to the expected socket, Promtail can’t connect.

Diagnosis: Check the status of the systemd-journald service:
```
sudo systemctl status systemd-journald
```
Also, verify the existence of the socket file:
```
ls -l /run/systemd/journal/stdout
```
The Fix: Ensure systemd-journald is active and running. If it’s not, start and enable it:
```
sudo systemctl start systemd-journald
sudo systemctl enable systemd-journald
```
If the stdout socket is missing, it might indicate a deeper systemd configuration issue, but usually, a restart of systemd-journald or a system reboot resolves this.
Why it Works: Promtail relies on the stdout socket provided by systemd-journald for its direct streaming capabilities. If this socket isn’t available, Promtail cannot establish the necessary communication channel.

4. Promtail Not Running as a Systemd Service or Incorrect Service File

Running Promtail directly from the command line is fine for testing, but for production, it should be a systemd service. If the service file is misconfigured, it can lead to permission or startup issues.

Diagnosis:
```
sudo systemctl status promtail
journalctl -u promtail
```
Look for errors related to file access, missing libraries, or network binding.
The Fix: Ensure your promtail.service file is correctly set up. A minimal example might look like this:
```
[Unit]
Description=Promtail Service
After=network.target

[Service]
User=promtail # Or another unprivileged user
Group=promtail
ExecStart=/usr/local/bin/promtail -config.file /etc/promtail/promtail-config.yaml
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
```
Make sure the User and Group exist and have appropriate permissions for their data directory and the config file. If you added capabilities, ensure they are in the [Service] section as noted in point 2.
Why it Works: A properly configured systemd service ensures Promtail starts automatically, runs with the correct user and group, and has its environment (like capabilities) set up correctly for reliable operation.

5. Promtail Version Compatibility with Journald API

While less common, very old versions of Promtail might have issues with newer journald features or API changes.

Diagnosis: Check the version of Promtail you are running:
```
promtail --version
```
Compare this with the release notes or known issues for the version of your operating system’s systemd.
The Fix: Update Promtail to the latest stable version. Download the latest binary from the Grafana Loki releases page on GitHub.
Why it Works: Newer versions of Promtail are tested and maintained against current systemd versions, ensuring compatibility with the journald API and its socket communication.

6. SELinux/AppArmor Restrictions

Mandatory Access Control systems like SELinux or AppArmor can prevent Promtail from accessing the journald socket even if file permissions and capabilities seem correct.

Diagnosis: Check the audit logs for SELinux or AppArmor denials related to promtail:

sudo ausearch -m avc -ts recent | grep promtail
# or for AppArmor
sudo dmesg | grep -i apparmor | grep promtail

The Fix (SELinux example): If you find denials, you’ll need to create or modify SELinux policies. A common fix is to allow Promtail to access journald:

# Example: Allow Promtail to read journald socket
sudo semanage fcontext -a -t systemd_journal_socket_t "/run/systemd/journal/stdout"
sudo restorecon -Rv /run/systemd/journal/stdout
# You might also need to adjust Promtail's type, e.g.,
# sudo semanage fcontext -a -t promtail_exec_t "/usr/local/bin/promtail"
# sudo restorecon -Rv /usr/local/bin/promtail

Consult SELinux or AppArmor documentation for specific policy creation.

Why it Works: SELinux/AppArmor policies explicitly define what processes can access which resources. Correctly configured policies grant Promtail the necessary permissions to interact with the journald socket and other required files.

After addressing these, the next error you might encounter if you’re using a distributed Loki setup is Promtail failing to connect to the Loki API endpoint due to network configuration or firewall issues.

1. Incorrect journald Path in Promtail Configuration

2. Missing CAP_DAC_READ_SEARCH Capability for Promtail