Loki’s RFC5424 parser is a bit of a black box until you realize it’s not trying to be a full syslog daemon, but rather a specialized tool for extracting specific fields from a structured log format.
Let’s see it in action. Imagine you have a Go application that’s spitting out RFC5424-formatted logs and you’re shipping them to Loki via Promtail. Here’s a snippet of what that log might look like:
<165>1 2023-10-27T10:30:00.123Z myhost appname procid msgid - - - This is the actual log message.
Promtail is configured to send this to Loki. The key here is how Loki’s parser, when enabled, will slice and dice this line. It doesn’t just store the whole string. It attempts to break it down into its constituent RFC5424 fields.
The RFC5424 format, as you know, is quite structured:
<PRI>VERSION TIMESTAMP HOSTNAME APP-NAME PROCID MSGID STRUCTURED-DATA MSG
Loki’s parser, when configured, will attempt to extract these into distinct labels or fields. The critical part is that it doesn’t do this by default. You need to tell it to.
Here’s how you’d enable and configure the RFC5424 parser in your Promtail scrape_configs for a specific job:
scrape_configs:
- job_name: my-rfc5424-logs
static_configs:
- targets:
- localhost
labels:
job: my-rfc5424-logs
__path__: /var/log/myapp.log
pipeline_stages:
- syslog:
# This is the magic. It tells Promtail to apply RFC5424 parsing.
# It will try to extract fields based on the RFC.
rfc5424: true
- labels:
# Now, map the extracted RFC5424 fields to Loki labels.
# These are the standard RFC5424 field names.
facility:
severity:
version:
timestamp:
hostname:
appname:
procid:
msgid:
- timestamp:
# Specify how to parse the timestamp field for Loki's internal indexing.
source: timestamp
format: RFC3339Nano
- output:
# Optionally, you can also extract the message part as a field.
source: msg
With this configuration, when Promtail reads /var/log/myapp.log, it sees the RFC5424-formatted lines. The syslog stage with rfc5424: true kicks in. It parses the line and makes the extracted fields available. The labels stage then explicitly maps these parsed fields (facility, severity, version, timestamp, hostname, appname, procid, msgid) into Loki labels. The timestamp stage tells Loki how to interpret the extracted timestamp for efficient querying. Finally, the output stage can capture the actual log message content into a field named msg.
Now, when you query Loki, you can do things like:
{job="my-rfc5424-logs", appname="appname", severity="error"}
Or even filter by specific message IDs:
{job="my-rfc5424-logs", msgid="MY-SPECIFIC-ID"}
This allows you to leverage the structured nature of RFC5424 logs directly within Loki’s query language, turning what would otherwise be a single, opaque string into rich, queryable metadata. The syslog stage in Promtail is the gateway; the labels stage is where you define what parts of that parsed syslog message become searchable labels in Loki.
The surprising thing about the syslog stage in Promtail is that it doesn’t use a full-blown syslog parser by default. When you set rfc5424: true, it’s applying a specific, optimized set of regexes and logic tailored only to the RFC5424 structure. It doesn’t attempt to handle older RFC3164 or other variations unless you explicitly configure separate stages for them. This specificity is what makes it performant but also means it’s less flexible than a general-purpose syslog daemon.
This configuration means that the facility and severity values are extracted from the initial <PRI> field. For <165>, the priority value is 165. This breaks down into facility 20 (local0) and severity 5 (notice). Promtail’s syslog stage handles this conversion automatically when rfc5424: true is set. You don’t need to manually decode the PRI value yourself.
The next step you’ll likely encounter is dealing with structured data fields within the RFC5424 message itself, which requires a different Promtail stage entirely.