In JMH(Java Microbenchmark Harness), we can use
@BenchmarkMode(Mode.AverageTime)
@Warmup(iterations = 10)
@Measurement(iterations = 10)
to evaluate the average time of an execution after JVM warms up.
Also we can use
@BenchmarkMode(Mode.SingleShotTime)
@Measurement(iterations = 1)
to estimate the cold start time of an execution. But this only executes the benchmark once, which may introduce bias. So is there any method to evaluate the average time of the cold start in JMH?
According to Alexey himself (though from 2014):
Single-shot benchmarks were originally destined to run a single measurement iteration over multiple forks -- the scenarios to estimate "cold" performance. But for many cases, you might want more measurement iterations there especially if you are running only a single fork, because more samples would be generated.
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
public class AverageSingleShot {
public static void main(String[] args) throws Exception {
Options opt = new OptionsBuilder()
.include(AverageSingleShot.class.getSimpleName())
.build();
new Runner(opt).run();
}
@Fork(100)
@Benchmark
@BenchmarkMode(Mode.SingleShotTime)
public int test() {
return ThreadLocalRandom.current().nextInt() + ThreadLocalRandom.current().nextInt();
}
}
Besides the fact that this will tell you the average (see that 100):
Benchmark Mode Cnt Score Error Units
AverageSingleShot.test ss 100 41173.540 ± 2871.546 ns/op
you will also get Percentiles and a Histogram.