How to run methods in benchmarks sequentially with JMH?

Question

In my scenario, the methods in benchmark should run sequentially in one thread and modify the state in order.

For example, there is a List called num in the benchmark class. What I want is: first, run add() to append a number into the list. Then, run remove() to remove the number from the list.

The calling sequence must be add() --> remove(). If remove() runs before add() or they run concurrently, they would raise exceptions because there's no element in the list.

That is, add() and remove() must be called sequentially and in one thread.

In Control the order of methods using JMH, I learned that the methods run in the lexicographical order. I tried the code below:

@State(Scope.Group)
@OutputTimeUnit(TimeUnit.MILLISECONDS)
@BenchmarkMode(Mode.AverageTime)
@Fork(value = 10)
public class ListBenchmark {

    private List num;

    public static void main(String[] args) throws RunnerException {
        Options options = new OptionsBuilder()
                .verbosity(VerboseMode.NORMAL)
                .syncIterations(true)
                .threads(1)
                .include(".*" + ListBenchmark.class.getCanonicalName() + ".*")
                .build();

        new Runner(options).run();
    }

    @Setup(Level.Invocation)
    public void setup() throws Exception {
        num = new ArrayList<>();
    }

    @Benchmark
    @BenchmarkMode(Mode.SingleShotTime)
    @Group("num")
    public void add() throws Exception {
        num.add(1);
    }

    @Benchmark
    @BenchmarkMode(Mode.SingleShotTime)
    @Group("num")
    public void remove() throws Exception {
        num.remove(0);
    }
}

But it doesn't work, because the add method and the remove method run concurrently. In some cases, the remove runs before add and raises IndexOutOfBoundsException.

How to run the methods in benchmarks sequentially with JMH?

user_1059372 · Answer

You start with a wrong precondition and everything fails short because of that. You can see a broader explanation from the authors here. You want symmetry where asymmetry is implied.

If you want to see how much it takes add -> remove place them both in the same @Benchmark, and same for individual add or remove via different State. For example:

@State(Scope.Thread)
public static class BothAddAndRemove {

    List num;

    @Setup(Level.Invocation)
    public void setup() throws Exception {
        num = new ArrayList<>();
    }
}

@State(Scope.Thread)
public static class RemoveOnly {

    List num;

    @Setup(Level.Invocation)
    public void setup() throws Exception {
        num = new ArrayList<>();
        num.add(1);
    }
}


@Fork(25)
@Benchmark
@BenchmarkMode(Mode.SingleShotTime)
public int add(BothAddAndRemove both) {
    both.num.add(1);
    return both.num.remove(0);
}

@Fork(25)
@Benchmark
@BenchmarkMode(Mode.SingleShotTime)
public int removeOnly(RemoveOnly removeOnly) {
    return removeOnly.num.remove(0);
}

How to run methods in benchmarks sequentially with JMH?

1 Answers