Does Python's multiprocessing package's spawning use the file state when the parent process started or the file state at the time of process spawning?

user_4084900 0
Viewed 797

If I have some Python code with a long setup stage, which eventually spawns processes, will the spawned process be based on the Python files at the start of the parent process or the files at the time of spawning?

That is, I start the parent Python process. Then I go and edit the Python files and finish editing them before the parent process spawns other processes. Finally, the parent process spawns child processes which use code from those files. Will the child processes use the code as it was when the parent started? Or the code as it was at the time of the process spawning?

python-multiprocessing python
1 Answers

The answer to your question depends on the operating system.

In Linux multiprocessing uses fork system call to create child processes. As a result, the child process "inherits" the bytecode of all python sources and does not re-read the sources. That is, in Linux the child will not recognize the changes.

In Windows multiprocessing creates child processes using _winapi.CreateProcess. The child initialises itself from scratch, that is it will read all source files again, including the modified ones.

Proof.

Here is a small example where the main processes modifies one of the source files.

somelib.py: prints out the process ID of the process that loads it.

import os

print("SomeLib loaded in process", os.getpid())

test.py: before spawning the child process it patches somelib.py

from multiprocessing import Process
# somelib prints process ID and, if patched, an extra line
import somelib

def f(name):
    print('hello', name)

if __name__ == '__main__':
    print("The main process is patching SomeLib")
    with open("somelib.py", "a+") as patch:
        patch.write("\n\nprint('SomeLib is patched')")
    p = Process(target=f, args=('bob',))
    # Spawn the the child
    p.start()
    p.join()

The output in Linux:

SomeLib loaded in process 70
The main process is patching SomeLib
hello bob

somelib.py was modified, but the child ignored that because of fork.

The output in Windows

SomeLib loaded in process 22512
The main process is patching SomeLib
SomeLib loaded in process 17008
SomeLib is patched
hello bob

See? The child with pid 17008 "re-loaded" somelib.py and processed the modified one.

user_90475890
Related