Infinite Tracing works by sampling a tiny fraction of your traces and storing them indefinitely, giving you a historical view of your application’s performance.
Let’s see it in action. Imagine a user request hitting your web server. New Relic’s APM agent, running alongside your application code, intercepts this request. It generates a trace, a detailed record of the journey this request takes through your system. This trace includes every service it touches, every database query, every external API call, and the time spent in each.
{
"traceId": "a1b2c3d4e5f67890",
"spans": [
{
"traceId": "a1b2c3d4e5f67890",
"spanId": "001",
"parentId": null,
"name": "WebTransaction/HTTP/GET/my-app/users",
"startTime": "2023-10-27T10:00:00Z",
"duration": 150, // milliseconds
"attributes": {
"service.name": "my-app",
"http.method": "GET",
"http.url": "/users"
}
},
{
"traceId": "a1b2c3d4e5f67890",
"spanId": "002",
"parentId": "001",
"name": "External/my-database/query",
"startTime": "2023-10-27T10:00:00.050Z",
"duration": 80, // milliseconds
"attributes": {
"db.system": "postgresql",
"db.statement": "SELECT * FROM users WHERE id = 123"
}
},
{
"traceId": "a1b2c3d4e5f67890",
"spanId": "003",
"parentId": "001",
"name": "External/auth-service/request",
"startTime": "2023-10-27T10:00:00.100Z",
"duration": 50, // milliseconds
"attributes": {
"http.method": "POST",
"http.url": "https://auth.example.com/verify"
}
}
]
}
This JSON represents a simplified trace. The traceId uniquely identifies the entire request’s journey. Each span is an individual operation within that trace, like a web transaction, a database query, or an external service call. The parentId links spans together, forming the hierarchical structure of the trace.
New Relic’s Distributed Tracing (DT) collects these spans from all your services. By default, it samples a percentage of these traces to send to the New Relic platform for analysis. This sampling is crucial for managing the volume of data. However, with Infinite Tracing, New Relic stores a small, configurable percentage of all your traces indefinitely, not just for the standard retention period (which is typically 15 days for traces). This means you can go back months, even years, to investigate historical performance issues.
The problem Infinite Tracing solves is the limitations of standard trace retention. If a performance degradation occurred three weeks ago, and your standard trace retention is 15 days, that data is gone. Infinite Tracing provides that long-term historical context.
Here’s how you configure it. You’ll need to set the infinite_tracing feature flag in your newrelic.yml configuration file.
common: &common
license_key: YOUR_LICENSE_KEY
app_name: YourAppName
<<: *common
# ... other configurations ...
infinite_tracing:
enabled: true
You also need to ensure your APM agents are configured to send trace data. For most agents, this is enabled by default when APM is configured. The key is to set the sampling rate for traces. While Infinite Tracing stores a percentage of traces indefinitely, the initial sampling rate determines how many traces are even considered for infinite storage. A lower sampling rate reduces data volume but might miss rare, intermittent issues. A higher rate captures more but increases costs. You control this via the sampling_rate setting in your newrelic.yml:
common: &common
license_key: YOUR_LICENSE_KEY
app_name: YourAppName
# Set sampling_rate to a value between 0.0 and 1.0.
# 0.1 means 10% of traces will be sampled.
sampling_rate: 0.1
<<: *common
# ... other configurations ...
infinite_tracing:
enabled: true
The sampling_rate attribute directly influences which traces are sent to New Relic for both standard retention and infinite tracing. If a trace isn’t sampled initially, it won’t be available for infinite tracing, regardless of the infinite_tracing.enabled flag. The infinite_tracing.enabled flag simply tells New Relic to keep the sampled traces for an extended period.
When you’re debugging an issue, you’ll navigate to the "Traces" section in the New Relic UI. You can then filter by time range, service, transaction name, or even specific attributes. Infinite Tracing expands your available time range significantly. You might then see a slow transaction from weeks ago that you can now investigate by drilling down into its spans.
The most surprising aspect of Infinite Tracing is that it doesn’t require a separate agent or a fundamentally different instrumentation. It leverages the existing distributed tracing data collected by your APM agents. The "magic" is in New Relic’s backend processing and storage, where a configured percentage of the sampled traces are shunted into a long-term, cost-effective storage solution, distinct from the hot storage used for standard retention. This allows for historical deep dives without overwhelming your immediate data ingestion pipeline.
Once you have Infinite Tracing set up and historical data available, the next logical step is to correlate this trace data with other observability signals, like logs and metrics, to build a more complete picture of past events.