Is it reproducible to use Arbitrary.sample from within an Action?

user_53615 0
Viewed 158

We have a stateful test for an order system. There is an Arbitrary that will generate an Order object that has a number of LineItem's.

There are actions to:

  • Create an Order
  • Cancel a LineItem

The action to create an order takes the order itself, eg:

Arbitraries.defaultFor(Order.class).map(CreateOrderAction::new)

The state for the actions has knowledge about all created orders.

To cancel a LineItem, we need knowledge about what orders are created. Inside CancelLineItemAction is it safe to do the following?

LineItem line = Arbitraries.<Collection<Order>>of(state.orders())
                           .flatMap(order -> Arbitraries.<Collection<LineItem>>of(order.lineItems()))
                           .sample();

Based on the javadoc of Arbitrary.sample(), it seems safe, but this construct isn't explicitly mentioned in the documentation on stateful tests, and we don't want to use it extensively only to break the reproducibility of our tests.

jqwik property-based-testing java
1 Answers

TLDR

  • Arbitrary.sample() is not designed to be used in that way
  • I recommend to use a random cancel index with modulo over the number of line items

1. Why Arbitrary.sample() is not recommended

Arbitrary.sample() is designed to be used outside of properties, e.g. to experiment with generated values or to use it in other contexts like JUnit Jupiter. There are at least three reasons:

  • The underlying random seed used for generating values depends on what happens before sampling. Thus the results are not really reproducible.
  • Sampling will not consider any added domain contexts that may change what's being generated.
  • Values generated by sample() DO NOT PARTICIPATE IN SHRINKING

2. Option 1: Hand in a Random object and use it for generating

Hand in a Random instance when generating a CancelLineItemAction:

Arbitraries.random().map(random -> new CancelLineItemAction(random))

Use the random to invoke a generator:

LineItem line = Arbitraries.of(state.orders())
           .flatMap(order -> Arbitraries.of(order.lineItems()))
           .generator(100).next(random).value();

But actually that's very involved for what you want to do. Here's a simplification:

3. Option 2: Hand in a Random object and use it for picking a line item

Same as above but don't take a detour with sampling:

List<LineItem> lineItems = state.orders().stream()
                                .flatMap(order -> order.lineItems().stream())
                                .collect(Collectors.toList());

int randomIndex = random.nextInt(lineItems.size());
LineItem line = lineItems.get(randomIndex);

Both option 1 and 2 will (hopefully) behave reasonably in jqwik's lifecycle but they won't attempt any shrinking. That's why I recommend the next option.

4. Option 3: Hand in a cancel index and modulo it over the number of line items

To generate the action:

Arbitraries.integer().between(0, MAX_LINE_ITEMS)
                     .map(cancelIndex -> new CancelLineItemAction(cancelIndex))

Use it in action:

List<LineItem> lineItems = state.orders().stream()
                                .flatMap(order -> order.lineItems().stream())
                                .collect(Collectors.toList());

int randomIndex = cancelIndex % lineItems.size();
LineItem line = lineItems.get(randomIndex);

The approach is described in more detail here: https://blog.johanneslink.net/2020/03/11/model-based-testing/

5. Future Outlook

In some more or less distant future jqwik may allow to hand in the current state when generating actions. This would make stuff like yours a bit simpler. But this feature has not yet been prioritized.

user_323520
Related