Promtail, the agent for Loki, can indeed slurp up Windows Event Logs, but getting it to do so reliably without dropping crucial information is trickier than it looks.
Collecting Windows Event Logs with Promtail
The most surprising thing about collecting Windows Event Logs with Promtail is how much of the context you lose if you’re not careful about structuring the data.
Let’s see it in action. Imagine you have a Windows machine and you want to capture Security event logs, specifically logon failures.
First, you need Promtail installed on your Windows machine. Then, you configure promtail-local-config.yaml.
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: windows_event_logs
static_configs:
- targets:
- localhost
labels:
job: windows_event_logs
__path__: "win-eventlog://{{ .System.ProviderName | regex `^.*$` }}/{{ .System.EventID | regex `^.*$` }}/{{ .System.Level }}/{{ .System.Channel }}/{{ .TimeCreated.SystemTime | time_format "%Y-%m-%d" }}/{{ .TimeCreated.SystemTime | time_format "%H" }}/{{ .TimeCreated.SystemTime | time_format "%M" }}/{{ .TimeCreated.SystemTime | time_format "%S" }}/"
windows_eventlog:
name: Security
event_id_include:
- 4625 # Logon failure
- 4624 # Logon success
# You can filter by level too, e.g., level_include: [ "Error", "Warning" ]
# Or by channel, e.g., channel_include: [ "System" ]
The __path__ field here is doing a lot of heavy lifting. It’s not a file path, but a directive to Promtail on how to structure the labels for Windows Event Logs.
win-eventlog://: This is a special scheme that tells Promtail to use the Windows Event Log API.{{ .System.ProviderName | regex^.$}}: This extracts theProviderNamefrom the event’sSystemdata and uses it as a label.regex \^.$`` is a no-op, simply ensuring the value is captured.{{ .System.EventID | regex^.*$}}: Extracts theEventID.{{ .System.Level }}: Extracts the logLevel(e.g., "Information", "Warning", "Error").{{ .System.Channel }}: Extracts theChannel(e.g., "Security", "System", "Application").{{ .TimeCreated.SystemTime | time_format "%Y-%m-%d" }}/{{ .TimeCreated.SystemTime | time_format "%H" }}/{{ .TimeCreated.SystemTime | time_format "%M" }}/{{ .TimeCreated.SystemTime | time_format "%S" }}: This is crucial for time-based partitioning in Loki. It breaks down the event’s timestamp into year, month, day, hour, minute, and second, creating a hierarchical path. This significantly improves query performance in Loki by allowing it to quickly prune irrelevant chunks.
The actual log content is sent as the log line itself. Promtail can also be configured to add extracted fields from the event data into the log line’s JSON payload, which is highly recommended for rich querying within Loki.
scrape_configs:
- job_name: windows_event_logs_with_fields
static_configs:
- targets:
- localhost
labels:
job: windows_event_logs
__path__: "win-eventlog://{{ .System.ProviderName | regex `^.*$` }}/{{ .System.EventID | regex `^.*$` }}/{{ .System.Level }}/{{ .System.Channel }}/{{ .TimeCreated.SystemTime | time_format "%Y-%m-%d" }}/{{ .TimeCreated.SystemTime | time_format "%H" }}/{{ .TimeCreated.SystemTime | time_format "%M" }}/{{ .TimeCreated.SystemTime | time_format "%S" }}/"
windows_eventlog:
name: Security
event_id_include:
- 4625
pipeline_stages:
- match:
selector: '{job="windows_event_logs"}'
stages:
- regex:
expression: '(?P<user>\S+) from (?P<source_ip>\S+) to (?P<target_user>\S+) with logon type (?P<logon_type>\d+)'
source: 'message' # The raw event message
- labels:
user:
source_ip:
logon_type:
- timestamp:
source: timecreated
format: RFC3339Nano # Or whatever format matches your event's timecreated
In this enhanced configuration, pipeline_stages are used to process the event message.
- The
regexstage parses themessagefield (which contains the human-readable event details) to extract specific pieces of information like the username, source IP, and logon type. - The
labelsstage then takes these extracted fields and turns them into Loki labels. This is powerful because it means you can querysource_ip="192.168.1.100"directly in Loki. - The
timestampstage can be used to align Loki’s timestamp with the event’s actualTimeCreatedfield, ensuring accurate ordering and retrieval.
When you query in Loki, you might use something like:
{job="windows_event_logs", eventid="4625"}
Or, with the extracted labels:
{job="windows_event_logs", eventid="4625", source_ip="192.168.1.100"}
The __path__ configuration is key because it dictates how Loki partitions your data. A well-structured __path__ that includes temporal components (YYYY-MM-DD/HH/MM/SS) allows Loki to efficiently locate data for specific time ranges. Without this, Loki would have to scan much larger chunks of data, severely impacting query performance. The windows_eventlog configuration block specifies which logs to read and can include/exclude specific event IDs or levels.
The most impactful element of this setup is the __path__ directive. It’s not just for file discovery; it’s Promtail’s primary mechanism for defining the label structure for the scraped data. For Windows Event Logs, using the win-eventlog:// scheme and then parsing event metadata (.System.ProviderName, .System.EventID, etc.) into labels is the standard way to get structured data into Loki. The temporal components within the __path__ are critical for Loki’s chunking strategy, making queries faster by enabling it to prune data based on time.
Once you have this set up, the next logical step is to explore how to correlate these Windows events with logs from other sources, like Linux syslog or application logs, within Loki.