JMH is designed to isolate code execution so effectively that it can sometimes measure the overhead of not running your code.
Let’s see JMH in action. Imagine we have a simple add method and we want to benchmark it.
public class MyBenchmark {
@State(Scope.Thread)
public static class BenchmarkState {
public int a = 10;
public int b = 20;
}
@Benchmark
public int add(BenchmarkState state) {
return state.a + state.b;
}
public static void main(String[] args) throws Exception {
org.openjdk.jmh.Main.main(args);
}
}
To run this, you’d typically compile it, package it into a JAR, and then execute it with JMH’s runner. The BenchmarkState class, annotated with @State(Scope.Thread), provides the input data for our benchmark method (add). JMH ensures that each thread running the benchmark gets its own instance of BenchmarkState. The @Benchmark annotation tells JMH which method to measure.
When you run this (e.g., java -jar benchmark.jar com.example.MyBenchmark), JMH does a lot behind the scenes. It warms up the JVM, runs your benchmark method thousands of times, and then measures the execution time. It handles things like dead code elimination, JIT compilation, and garbage collection pauses to give you a reliable measurement. The output will show you metrics like average time per operation, throughput, and standard deviation, giving you confidence that the numbers reflect your code’s actual performance, not the testing framework’s.
The core problem JMH solves is the inherent difficulty of accurately measuring code performance in a modern, highly optimized JVM. Simple System.nanoTime() loops are notoriously unreliable because they don’t account for:
- JIT Compilation: The JVM optimizes code during execution. A piece of code run for the first time might be slow, but after a few warm-up runs, the JIT compiler will generate highly optimized machine code. JMH’s warm-up phases ensure your code is fully optimized before measurements begin.
- Dead Code Elimination: If a compiler or the JVM determines that the result of a computation is never used, it might optimize that computation away entirely. JMH uses "black hole" methods (like
org.openjdk.jmh.infra.Blackhole) to consume the results of your benchmarked code, preventing dead code elimination. - Garbage Collection: GC pauses can significantly skew performance measurements. JMH attempts to run benchmarks for long enough that GC activity is amortized, and it can even report GC statistics.
- Concurrency and Threading: Real-world applications are often multi-threaded. JMH allows you to configure how many threads participate in the benchmark and how state is shared (or not shared) between them.
- Inlining: Method calls have overhead. The JIT compiler often inlines small methods, effectively merging their code. JMH’s warm-up and measurement phases allow the JIT to decide whether to inline your benchmarked methods.
The key levers you control in JMH are primarily through annotations and configuration properties. You define @State classes to manage test data and its scope (Thread, Benchmark, Group, Global). You annotate methods with @Benchmark and control the benchmark mode (Throughput, AverageTime, SampleTime, SingleShotTime) and the number of forks (@Fork) and warm-up iterations (@Warmup). The Options builder in the Main.main method is where you set these parameters programmatically.
The one thing most people don’t realize is that JMH’s @State scopes are not just about how state is initialized, but also about how JMH manages threads around that state. For Scope.Thread, JMH creates a new instance of the state for each thread that will be used for the benchmark. If you use Scope.Benchmark, there’s one instance per benchmark run, shared across all threads for that run. If you choose Scope.Group, it’s one instance per thread group, and Scope.Global is one instance for the entire benchmark execution across all threads and forks. This distinction is critical for understanding how contention (or lack thereof) affects your measurements.
The next logical step after mastering basic benchmarking is exploring JMH’s advanced sampling and profiler integration.