Alloy doesn’t send logs to Loki; it ingests logs from various sources, processes them, and then exports them to destinations like Loki.
Here’s how to set up Alloy to collect logs using OpenTelemetry and send them to Loki:
Imagine you have an application generating logs. You want to get those logs into Loki for analysis. OpenTelemetry is the standardized way to describe and generate telemetry data (logs, metrics, traces), and Alloy is a powerful agent that can consume this data, transform it, and send it to your observability backend.
Let’s set up a basic Alloy configuration to achieve this.
1. The Alloy Configuration (alloy.alloy)
logging {
level = "info"
}
// Define the source of your logs. This example uses a file.
// In a real-world scenario, this could be journald, docker, etc.
file_source "my_app_logs" {
path = "/var/log/my_application.log"
// Use the OTLP log receiver to parse log lines as OTLP log records.
parser {
otlp_log {}
}
}
// Define the OpenTelemetry Collector component for processing.
// This is where you'd add processors if needed (e.g., batching, filtering).
otelcol_receiver "otlp" {
// Alloy will automatically expose an OTLP receiver endpoint.
// The logs will be available at this receiver for other components to consume.
}
// Define the exporter to send logs to Loki.
loki_exporter "my_loki" {
endpoint = "http://loki:3100/loki/api/v1/push" // Replace with your Loki endpoint
// Add labels to your logs in Loki.
// This is crucial for filtering and querying.
labels = {
job = "my-app",
environment = "production",
}
}
// Connect the components:
// - Route logs from the file_source to the otelcol_receiver.
// - Route logs from the otelcol_receiver to the loki_exporter.
// The "logs" component is how Alloy handles log processing pipelines.
logs {
// Ingest logs from the file source.
// The `file_source` automatically sends its output to the `otelcol_receiver`
// if it's the only receiver defined in the `logs` block.
// For more complex setups, you'd explicitly define `receivers = [...]`.
// Export logs to Loki.
// The `otelcol_receiver` makes logs available for export.
// The `loki_exporter` consumes these logs.
exporters = [ "my_loki" ]
}
// Run the Alloy agent with this configuration.
// This tells Alloy to start the defined components and pipelines.
run {
// The component that acts as the entry point for log collection.
// It will listen for logs from configured sources and send them to exporters.
logs = [
"file_source.my_app_logs",
"otelcol_receiver.otlp", // Although not explicitly receiving, it's part of the processing chain.
]
}
Explanation:
file_source "my_app_logs": This component watches a specific file (/var/log/my_application.log) for new log entries. Theotlp_log {}parser within it is key: it expects the log file to contain entries formatted as OpenTelemetry Log Records. If your application doesn’t directly output OTLP, you’d use a different parser (likejson_parser,regex_parser, etc.) and then potentially a processor to convert it to OTLP if needed for downstream components.otelcol_receiver "otlp": This is a placeholder for the OpenTelemetry Collector’s OTLP receiver. Alloy, when configured withotelcol_receiver, provides an endpoint that can receive OTLP data. In this setup,file_sourceis implicitly sending its parsed OTLP logs to this receiver.loki_exporter "my_loki": This is where the magic happens for Loki. It takes the OTLP log records it receives and pushes them to your Loki instance at the specifiedendpoint. Thelabelsare crucial; they become the metadata associated with your logs in Loki, allowing you to filter logs byjoborenvironment.logs {}block: This is Alloy’s way of defining a log processing pipeline. It specifies which receivers (sources) feed into which exporters (destinations). Here, the logs fromfile_source.my_app_logsare processed and then sent toloki_exporter.my_loki.run {}block: This tells Alloy which components to actually start and manage. We list our log source and the OTLP receiver that acts as the intermediary.
How to Run This:
- Install Alloy: Download and install the Alloy agent.
- Create
alloy.alloy: Save the configuration above into a file namedalloy.alloy. - Create Dummy Log File:
Note: For thisecho '{"severity": "info", "message": "Application started", "trace_id": "12345", "span_id": "67890"}' > /var/log/my_application.log echo '{"severity": "error", "message": "Failed to connect to database", "error_message": "Connection refused"}' >> /var/log/my_application.logotlp_logparser to work correctly, your log file entries must be valid OTLP Log Record JSON. If your application outputs plain text or different JSON structures, you’ll need to adjust theparserinfile_sourceaccordingly. - Start Alloy: Run Alloy using the configuration file:
./alloy --config-file alloy.alloy - Check Loki: After a few moments, you should see logs appearing in Loki, queryable by
job="my-app"andenvironment="production".
The Surprising Part: OTLP as the Universal Log Format
The most counterintuitive aspect of this setup is that Alloy doesn’t inherently know about "application logs" or "system logs." It operates on the OpenTelemetry Protocol (OTLP) for logs. When you configure a file_source with an otlp_log parser, you’re telling Alloy, "Treat the contents of this file as if they were already structured according to the OTLP Log Record specification." This means your application (or a preceding agent) must be capable of emitting logs in this format, or you need a parser in Alloy that can translate your application’s native log format into OTLP.
System in Action (Conceptual Flow):
- Application Writes Log: Your application writes a log entry.
- Log Appears in File: The log entry is appended to
/var/log/my_application.log. - Alloy
file_sourceDetects Change: Alloy’sfile_source.my_app_logswatches the file and reads the new line. otlp_logParser Processes Line: The parser attempts to interpret the line as an OTLP Log Record. If successful, it creates an OTLP Log Record object.- Data Flows to
otelcol_receiver: The parsed OTLP Log Record is passed to theotelcol_receiver. - Data Flows to
loki_exporter: Theotelcol_receivermakes the log available, and theloki_exporter.my_lokipicks it up. - Loki Exporter Pushes to Loki: The exporter formats the OTLP Log Record into a Loki-compatible push request and sends it to
http://loki:3100/loki/api/v1/push, adding the specifiedjobandenvironmentlabels. - Loki Stores Log: Loki receives the data and indexes it with the associated labels, making it available for querying.
The Mental Model:
Alloy acts as a highly configurable pipeline. You define the stages:
- Sources: Where data comes from (files, network ports, other agents).
- Receivers: How data is ingested into Alloy’s internal processing (often OTLP-based).
- Processors: Components that transform data (batching, filtering, adding attributes, sampling).
- Exporters: Where data goes (Loki, Prometheus, Kafka, etc.).
The logs {} block orchestrates which sources feed into which exporters through the internal receiver/processor chain. The run {} block tells Alloy which of these defined components to activate.
One Thing Most People Don’t Know:
The otlp_log parser is incredibly powerful but also very strict. It doesn’t just expect JSON; it expects JSON that conforms to the OpenTelemetry Log Record schema, including fields like severity, body, and optional attributes like trace_id, span_id, resource.attributes, and attributes. If your log lines are almost OTLP but missing a required field or have it in the wrong format, the otlp_log parser will fail, and the log won’t be ingested as an OTLP record. You’ll need to use a more flexible parser (like json_parser or regex_parser) and potentially a transform processor to massage the data into the correct OTLP structure before it reaches components that expect OTLP.
The next concept you’ll likely explore is how to handle logs that are not already in OTLP format, requiring custom parsing and transformation within Alloy.