You can link Loki logs to distributed traces, but it’s not about Loki generating traces. Instead, Loki consumes trace IDs from your applications and uses them to find related log lines.
Let’s see this in action. Imagine a user request hitting your service. This request is assigned a trace ID. As it flows through different microservices, each service logs events, and crucially, includes that same trace ID in its logs. Loki, when queried, can then filter logs based on this trace ID, effectively showing you the log stream for a specific transaction.
Here’s a simplified example of what those logs might look like in Loki. Notice the traceID field:
{app="frontend", level="info"} 2023-10-27T10:00:01Z "Received request" traceID="abc123xyz"
{app="user-service", level="info"} 2023-10-27T10:00:02Z "Processing user data" traceID="abc123xyz"
{app="order-service", level="error"} 2023-10-27T10:00:03Z "Failed to create order" traceID="abc123xyz"
{app="payment-service", level="info"} 2023-10-27T10:00:04Z "Payment processed successfully" traceID="abc123xyz"
When you’re looking at a trace in a tracing system (like Jaeger, Tempo, or Grafana’s Explore view when connected to a tracing backend), you’ll see the spans representing the journey of that request. Each span, in this setup, has a reference to its corresponding trace ID. You can then click on a span or the trace ID itself, and Loki will be queried to show you all logs that share that traceID.
The mental model here is that your tracing system provides the timeline and structure of a request’s journey across services, while Loki provides the detailed narrative within that timeline, filtered by the common thread of the trace ID.
The problem this solves is immense: when a distributed transaction fails, you don’t have to guess which service is misbehaving. You can see the trace, identify the problematic span (e.g., an error in the order-service), and then instantly pull up all the logs from order-service (and any other service involved in that specific trace) that occurred during that transaction. This drastically cuts down debugging time from hours to minutes.
To make this work, your applications need to:
- Generate or propagate trace IDs: When a request enters your system, a trace ID is generated. This ID must be passed along to downstream services, often via HTTP headers (like
traceparentfrom W3C Trace Context) or message queue metadata. - Include trace IDs in logs: Every log message generated by your services must include the current trace ID as a structured field. This is usually done via your logging library’s context propagation features. For example, in Go, you might use
zapcore.ObjectEncoderor similar to addtraceIDto every log record. In Java, libraries like Logback or Log4j can be configured to include MDC (Mapped Diagnostic Context) values liketraceID.
Here’s how you might configure Loki to ingest and query these logs. In your Loki configuration (loki-config.yaml), you’d have your ingestion pipeline. While Loki itself doesn’t parse trace IDs in a special way during ingestion, it indexes them as labels if you tell it to. However, the more common and powerful approach is to query them directly using LogQL.
Let’s say you’ve configured your Prometheus agents (like promtail) to scrape logs and are forwarding them to Loki. Your promtail configuration might look something like this for a specific service:
scrape_configs:
- job_name: myapp-logs
static_configs:
- targets:
- localhost
labels:
job: myapp
__path__: /var/log/myapp/*.log
# You can add static labels here that apply to all logs from this job
pipeline_stages:
# This stage assumes your logs are JSON formatted
- json:
expressions:
level: level
message: message
traceID: traceID # Loki will index 'traceID' as a label if it's present in the JSON
# If your logs are not JSON, you might use regex to extract
# - regex:
# expression: '(?P<time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z) (?P<level>\w+) (?P<message>.*) traceID=(?P<traceID>\w+)'
# - labels:
# traceID:
The key here is the pipeline_stages. If your logs are JSON, the json stage extracts fields. If traceID is present in the JSON, Loki will automatically index it as a label (if you don’t explicitly exclude it). If your logs are plain text, you’d use regex to extract the traceID and then the labels stage to make it a queryable label.
Once ingested, querying in Loki is straightforward. If you’re in Grafana’s Explore view, and you have Loki as a data source, you’ll see a query builder. If you’re using LogQL directly, you can filter by the trace ID label.
To find all logs for trace abc123xyz from the myapp job:
{job="myapp", traceID="abc123xyz"}
Or, if you want to see logs from a specific service within that trace:
{job="myapp", app="order-service", traceID="abc123xyz"}
The power comes when you integrate this with your tracing backend. Most modern observability platforms allow you to configure "traces-to-logs" correlation. In Grafana, when you’re viewing a trace, you can often click on a span, and it will automatically populate a Loki query in the log panel below, filtering for logs with the traceID and potentially other contextual labels (like service.name or span.id) from that specific span.
The one thing that often trips people up is that Loki itself doesn’t understand the concept of a "trace" or "span" natively. It just sees log lines with arbitrary labels. The correlation happens because your applications are diligently putting the same traceID into both their log output and the metadata that your tracing system uses. If your logging system isn’t configured to extract or index traceID as a label, or if your applications stop propagating the traceID correctly, the correlation breaks. You’ll see logs, but they won’t be linked to the trace anymore. It’s a convention and a shared piece of data, not a built-in protocol between Loki and your tracing system.
The next step after successfully correlating logs to traces is often implementing "logs-to-traces" correlation, where you can click on a specific log line and jump to the relevant span in your tracing system.