Write to same file in pytest multiprocessing

user_13421357 0
Viewed 283

There is a pytest test suite that gets run in multiprocess/parallel via pytest-xdist. Some of these tests write to the same file, which can become problematic when multiprocessing.

I thought that doing something like this would do the trick, but hasn't been successful:

In util.py:

import multiprocessing

LOCK = multiprocessing.Lock()

And in test_something.py:

from util import LOCK
...
def test_something():
    ...
    LOCK.acquire()
    write_to_file()
    LOCK.release()
    ...

Sometimes the tests would hang, or sometimes there would be a WRITE/READ issue.

Am I placing the LOCK at an incorrect location? Is there a way to pass a global object across all of the tests? Or am I thinking about it the wrong way?

multiprocess pytest-xdist pytest python
1 Answers

For that, you'll probably want to use https://py-filelock.readthedocs.io/en/latest/

pip install filelock, then try the examples, eg:

from filelock import Timeout, FileLock

lock = FileLock("high_ground.txt.lock")
with lock:
    open("high_ground.txt", "a").write("You were the chosen one.")
user_70582660
Related