pytest-randomly - detect test coupling

user_6730526 0
Viewed 335

Summarize the problem:

I have ~1000 tests, tested with pytest-randomly. I know they fail due to coupling, when --randomly-seed=42, for Example:

test_1 OK
test_2 OK
...
test_x OK [the one I want to find]
...
test_y FAILED [ y and x are coupled ]
...
test_1000

I'd like to automatically find the smallest test sequence, that creates the coupling (for this example, it would be test_x -> test_y). Is there a way to do it?

What you've tried

  • Running the whole test suite until I found, that seed 42 creates an error.

  • I think this is solvable by running test cases 1000 times in --randomly-seed=42 order - each time excluding one test, and checking if tests still fail. If they fail, the excluded test is a noise, if they don't, the excluded test is necessary to reproduce the error. This way, after 1000 runs I should have a much smaller set of tests causing the error. But this sounds like something clearly already implemented.

  • Other problem would be - how to recreate the exact test ordering, but without one chosen test?

  • I am basically looking for something that would look through a given sequence... similar way git rerere goes through git commits.

pytest testing python
2 Answers

I haven't used pytest-randomly, but I can suggest an algorithm:

  1. Discard all tests run after the failing test.
  2. Split the data in half.
  3. Run the tests of each half, following each half with the failed test.
  4. If the failed test succeeds, then discard all tests in that half.
  5. Go to step 2.

It's essentially a binary search, but putting the failed test into the position where it would normally be. Once the scope of the binary search is narrowed to include only one element other than the failed test, you know that it is the coupled one.

user_55095750

To expand on the answer by @Hack5:
I thought that this is something that can be automated, so I wrote a small pytest plugin that does that. It starts by running tests twice in different directions (e.g. forwards and backwards) in separate test runs, and if it find tests that fail only in one of the runs, it goes on running subsets of the tests until it finds the dependency (using a kind of binary search that is a bit optimized compared to the proposed one).

Install it using

pip install pytest-find-dependencies

and use it like:

pytest --find-dependencies

You need some patience if you have a lot of tests...

Note that this works only if a test does not change the environment permanenently - the assumption is that the state is reset on each test run (which is also true for the other answer).

Note also that if you start it using the seed found by pytest-randomly, it will use this for the first run, but it will still run the tests backwards afterwards.

user_124807300
Related