Grafana’s Trace-to-Logs feature lets you instantly pivot from a distributed tracing span to the exact logs generated around that specific operation.

Let’s see it in action. Imagine you’re looking at a trace in Grafana, and you’ve identified a specific span that’s taking too long. This span represents a single request or operation within your distributed system, like GET /api/v1/users/{id}. You click on this span, and in the details panel, you see a new section: "Related Logs." Clicking that section takes you directly to your logs, filtered to show only the log lines that occurred during the exact time window and, crucially, are associated with that specific trace ID and span ID.

Here’s a simplified example of what that looks like in a Grafana explore view.

First, you have a trace, identified by a traceID. Within that trace, you have a specific spanID.

{
  "traceID": "a1b2c3d4e5f67890",
  "spanID": "abcdef0123456789",
  "operationName": "GET /api/v1/users/{id}",
  "startTime": "2023-10-27T10:00:00Z",
  "duration": 1500, // milliseconds
  "tags": {
    "http.method": "GET",
    "http.url": "/api/v1/users/123",
    "http.status_code": 200,
    "service.name": "user-service"
  }
}

Now, you want to find the logs for this. Trace-to-Logs configures your logging datasource to understand how to filter by traceID and spanID. When you click on the "Related Logs" button associated with the above span, Grafana constructs a query for your logging backend. If your logs are in Loki, it might look like this:

{service="user-service"} |= "traceID=a1b2c3d4e5f67890" |= "spanID=abcdef0123456789"

This query tells Loki to find log lines from the user-service that contain both the specific traceID and spanID. The result is a focused view of logs relevant only to that one operation, making debugging incredibly efficient.

The problem Trace-to-Logs solves is the immense difficulty of correlating events across distributed systems. When an error occurs, or a request is slow, you might see the symptom in a trace, but finding the cause requires sifting through potentially gigabytes of logs. Without this direct link, you’d have to manually:

  1. Extract the traceID from the trace span.
  2. Extract the spanID from the trace span.
  3. Get the timestamp of the span.
  4. Go to your logging system.
  5. Search for log lines containing the traceID within a small time window around the span’s timestamp.
  6. Hope that your logs are instrumented to include traceID and spanID.

Trace-to-Logs automates steps 1-5 and makes step 6 a configuration problem, not an instrumentation one.

The internal mechanism relies on a few key components:

  • Tracing Backend: Your distributed tracing system (e.g., Jaeger, Tempo, Zipkin) that stores trace data.
  • Logging Backend: Your log aggregation system (e.g., Loki, Elasticsearch, Splunk) that stores log data.
  • Grafana: The visualization layer that orchestrates the query.
  • Datasource Configuration: Grafana’s configuration for both your tracing and logging datasources. This is where you tell Grafana how to link them. Specifically, you define a "Trace to logs" mapping within the tracing datasource configuration. This mapping specifies how to extract traceID and spanID from trace spans and how to use them to construct a query for the logging datasource.

The "Trace to logs" mapping in your tracing datasource configuration in Grafana is critical. You’ll typically define a spanLinkTemplate. This template uses Go template syntax to dynamically build the URL or query for your logging datasource. For a Loki datasource, it might look something like this:

/explore?orgId=1&left=["now-1h","now","","{service=~\"$service\"} |= \"traceID=$traceID\" |= \"spanID=$spanID\""]

Here, $service, $traceID, and $spanID are variables Grafana extracts from the current trace span. When you click the "Related Logs" button, Grafana substitutes these variables and navigates you to the specified URL, effectively performing the filtered log query on your behalf.

The counterintuitive part of Trace-to-Logs is that it doesn’t require your logs to be explicitly instrumented with span IDs if your tracing system is configured correctly. Many tracing systems, when they generate spans, also generate corresponding log entries within the tracing system itself, or they use specific conventions. However, the most robust implementations rely on your application code adding traceID and spanID as fields or attributes to the log messages themselves. The Grafana configuration then becomes a sophisticated parser and filter, leveraging these fields in your log data, rather than relying on implicit relationships. The power comes from the consistency of how your application emits logs, ensuring that a traceID and spanID are present and correctly formatted for Grafana to pick up and use in its query.

The next logical step after correlating traces and logs is to understand how to automatically generate alerts based on patterns observed in these correlated events.

Want structured learning?

Take the full Grafana course →