The New Relic Java APM agent, when configured for deep tracing, doesn’t just sample transactions; it actively instruments your JVM to capture detailed method-level execution data for every request.
Let’s see this in action. Imagine a simple Spring Boot application.
@RestController
public class GreetingController {
@Autowired
private GreetingService greetingService;
@GetMapping("/hello")
public String sayHello(@RequestParam(value = "name", defaultValue = "World") String name) {
String greeting = greetingService.getGreeting(name);
return greeting + " from deep trace!";
}
}
@Service
public class GreetingService {
public String getGreeting(String name) {
// Simulate some work
try {
Thread.sleep(50);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
return String.format("Hello, %s", name);
}
}
Without deep tracing, you’d see a transaction for /hello, its total duration, and perhaps some database calls. With deep tracing enabled, you’d see exactly how much time was spent inside GreetingController.sayHello, then inside GreetingService.getGreeting, and even the Thread.sleep call itself, all broken down to the millisecond.
The core problem deep tracing solves is the "black box" nature of APM sampling. When sampling is used, New Relic only reports on a percentage of transactions. This means if a performance issue is intermittent or only affects a small subset of requests, you might never see it. Deep tracing, by instrumenting every method call within the JVM, provides a complete, albeit more verbose, picture.
Here’s how the New Relic Java agent achieves this:
- Bytecode Instrumentation: The agent attaches to your JVM and, at runtime, rewrites the bytecode of your application’s classes. It injects small pieces of code (probes) at the entry and exit points of methods. These probes record timestamps and other context.
- Trace Assembly: As requests flow through your application, these probes fire. The agent collects the data from these probes and assembles it into detailed traces, showing the call hierarchy and timing for each method within a transaction.
- Data Reporting: This granular trace data is then sent to the New Relic platform for analysis and visualization.
To configure deep tracing, you primarily interact with the newrelic.yml configuration file. The key setting is distributed_tracing.enabled and, crucially for deep method-level visibility, tracing.method_tracer.enabled.
The most surprising thing about deep tracing is that it can capture synchronous I/O operations as distinct spans, even if the underlying library doesn’t explicitly support asynchronous APIs. For example, a standard java.net.HttpURLConnection call, which is inherently blocking, will appear as a measurable span in New Relic’s deep traces, allowing you to isolate the time spent on network I/O versus application processing.
To enable deep tracing, you’ll typically set the following in your newrelic.yml:
# newrelic.yml
common: &default_config
# ... other common settings ...
app_name: MyJavaDeepTraceApp
license_key: YOUR_LICENSE_KEY
agent_config:
<<: *default_config
# Enable distributed tracing to link traces across services
distributed_tracing:
enabled: true
# Enable method tracer for deep tracing
tracing:
method_tracer:
enabled: true
# Optional: Control the maximum number of method traces per transaction
# This can prevent excessive memory usage in very high-throughput scenarios
max_method_traces_per_tx: 1000
# Optional: Set a threshold for tracing methods. Methods taking longer
# than this (in milliseconds) will always be traced.
trace_threshold: 5
After restarting your Java application with this configuration, requests to /hello will now show detailed breakdowns in the New Relic UI, including the Thread.sleep duration and the exact time spent in GreetingService.getGreeting.
The trace_threshold setting is particularly useful. Setting it to 5 means any method that takes longer than 5 milliseconds will be automatically included in the deep trace, even if it’s not explicitly sampled by other tracing rules. This helps focus on the "slow" parts without overwhelming you with data from every single tiny method call.
The next concept you’ll encounter is how to effectively filter and analyze the massive amount of data generated by deep tracing, particularly when dealing with distributed systems and asynchronous operations.