Is there a way to run several time a combinaison of python code and Pytest tests automatically?

user_14155793 0
Viewed 147

I am looking to automate the process where: I run some python code, then run a set of tests using pytest then, if all tests are validated, start the process again with new data.

I am thinking of writing a script executing the python code, then calling pytest using pytest.main(), check with the help of the exit code that all tests passed and in case of success start again.

The issue is that it is stated in pytest docs (https://docs.pytest.org/en/stable/usage.html) that it is not recommended to make multiple calls to pytest.main():

Note from pytest docs:

"Calling pytest.main() will result in importing your tests and any modules that they import. Due to the caching mechanism of python’s import system, making subsequent calls to pytest.main() from the same process will not reflect changes to those files between the calls. For this reason, making multiple calls to pytest.main() from the same process (in order to re-run tests, for example) is not recommended."

I was woundering if it was ok to call pytest.main() the way I intend to or if there was any better way to achieve what I am looking for?

I've made a simple example to make it problem more clear:

A = [0]

def some_action(x):  
    x[0] += 1

if __name__ == '__main__':
    print('Initial value of A: {}'.format(A))
    for i in range(10):
        if i == 5:
            # one test in test_mock2 that fails
            test_dir = "./tests/functional_tests/test_mock2.py"
        else:
            # two tests in test_mock that pass
            test_dir = "./tests/functional_tests/test_mock.py"
        some_action(A)
        check_tests = int(pytest.main(["-q", "--tb=no", test_dir]))
        if check_tests != 0:
            print('Interrupted at i={} because of tests failures'.format(i))
            break
    if i > 5:
        print('All tests validated, final value of A: {}'.format(A))
    else:
        print('final value of A: {}'.format(A))

In this example some_action is executed until i reaches 5, at which point the tests fail and the process of executing/testing is interrupted. It seems to work fine, I'm only concerned because of the comments in pytest docs as stated above

pytest python
1 Answers

The warning applies to the following sequence of events:

  1. Run pytest.main on some folder which imports a.py, directly or indirectly.
  2. Modify a.py (manually or programatically).
  3. Attempt to rerun pytest.main on the same directory in the same python process as #1

The second run in step #3 will not not see the changes you made to a.py in step #2. That is because python does not import a file twice. Instead, it will check if the file has an entry in sys.modules, and use that instead. It's what lets you import large libraries multiple times without incurring a huge penalty every time.

Modifying the values in imported modules is fine. Python binds names to references, so if you bind something (like a new integer value) to the right name, everyone will be able to see it. Your some_action function is a good example of this. Future tests will run with the modified value if they import your script as a module.

The reason that the caveat is there is that pytest is usually used to test code after it has been modified. The warning is simply telling you that if you modify your code, you need to start pytest.main in a new python process to see the changes.

Since you do not appear to be modifying the code of the files in your test and expecting the changes to show up, the caveat you cite does not apply to you. Keep doing what you are doing.

user_29887300
Related