JMH benchmarking avoid jvm optimization

user_3892213 0
Viewed 192

I am trying to write jmh benchmarks.

I am came across various blogs mentioning pitfalls in jmh benchmarking. The common example are

  1. this code
int sum() {
   int a =7;
   int b = 8;
   return a+b;
}

will be optimised to

int sum() {
return 15;
}
  1. this code
int sum(int y) {
   int x = new Object();
   return y;
}

will be optimised to

int sum(int y) {
   return y;
}

i.e. removing non used object initialisation.

But this list is no where extensive to what all sort of optimisation jvm will do.

Below is the problem i am facing.

Lets say there are few methods and here is how call graph looks like

int methodA(CustomObjectA a) {
   //do something 
   methodB(a);
   //do something
   return returnValueA;
}

int methodB(CustomObjectA a) {
   //do something 
   methodC(a);
   //do something
   return returnValueB;
}

int methodC(CustomObjectA a) {
   //do something
   return returnValueC;
}

And we will try to benchmark methodA. By Passing CustomObjectA created in state object. But

  1. from JVM perspective methodC is always called with same reference, wouldn't it optimise methodC to return same returnValueC all the time?

  2. Why would it not do this?

  3. How can we make sure this optimisation will not be done? by passing different reference every time using @State(Scope.Thread)?

  4. Is there any exhaustive list to explain what all optimisation might be possible?

jmh microbenchmark performance-testing java performance
1 Answers

Are you saying that you want to test methodA and all of those other methods are private and this is how the call chain looks like? If so, JMH here is irrelevant - what optimizations are going to be applied, will still be applied to that code. It is also quite impossible to tell what optimizations might happen in the end, since they are a lot on the JVM, and also depend on a lot of other factors like operating system, CPU, etc; so an "extensive list" simply can't exist.

Depending on what you do in that //do something in each method, that code could be elided or not, for example. Look at this simplified example:

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@Warmup(iterations = 5, time = 5)
@Measurement(iterations = 5, time = 5)
public class Sample {

    private static final int ME = 1;

    public static void main(String[] args) throws Exception {
        Options opt = new OptionsBuilder()
                .include(Sample.class.getSimpleName())
                .build();

        new Runner(opt).run();
    }

    @Benchmark
    public int methodOne(CustomObjectA a) {
        simulateWork();
        return 42;
    }

    @Benchmark
    public int methodTwo(CustomObjectA a, Blackhole bh) {
        bh.consume(simulateWork());
        return 42;
    }

    @State(Scope.Thread)
    public static class CustomObjectA {

    }

    private static double simulateWork() {
        return ME << 1;
    }

}

The difference is that in method methodTwo, I use a so called Blackhole (read this for more details), while in methodOne, I do not. As a result simulateWork is eliminated from methodOne, as the results show:

Benchmark         Mode  Cnt  Score   Error  Units
Sample.methodOne  avgt   25  1.950 ± 0.078  ns/op
Sample.methodTwo  avgt   25  3.955 ± 0.120  ns/op

On the other hand, if I even slightly change the code to have the smallest possible side effects:

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@Warmup(iterations = 5, time = 5)
@Measurement(iterations = 5, time = 5)
public class Sample {

    public static void main(String[] args) throws Exception {
        Options opt = new OptionsBuilder()
                .include(Sample.class.getSimpleName())
                .build();

        new Runner(opt).run();
    }

    @Benchmark
    public int methodOne(CustomObjectA a) {
        simulateWorkWithA(a);
        return 42;
    }

    @Benchmark
    public int methodTwo(CustomObjectA a) {
        return simulateWorkWithA(a) + 42;
    }

    @Benchmark
    public int methodThree(CustomObjectA a, Blackhole bh) {
        bh.consume(simulateWorkWithA(a));
        return 42;
    }

    @State(Scope.Thread)
    public static class CustomObjectA {
        int x = 0;
    }

    private static int simulateWorkWithA(CustomObjectA a) {
        return a.x = a.x + 1;
    }

}

the elimination of simulateWorkWithA(a) in methodOne will not happen:

Benchmark           Mode  Cnt  Score   Error  Units
Sample.methodOne    avgt   25  2.267 ± 0.198  ns/op
Sample.methodThree  avgt   25  3.711 ± 0.131  ns/op
Sample.methodTwo    avgt   25  2.325 ± 0.008  ns/op

Notice that there is almost no difference between methodOne and methodTwo.

user_10593720
Related