Possible to create Python multiprocessing child/worker processes that do not import the __main__ module?

user_6815395 0
Viewed 346

The standard behavior of multiprocessing on Windows is to import the __main__ module into child processes when spawned.

For large projects with many imports, this can significantly slow down the child process startup, not to mention the extra resources consumed. It seems very inefficient for cases where the child process will run a self-contained task that only uses a small subset of those imports.

Is there a way to explicitly specify the imports for the child processes? If not the multiprocessing library, is there an alternative?

While I'm specifically interested in Python 3, answers for Python 2 may be useful for others.

Edit

I've confirmed that the approach suggested by Lie Ryan works, as shown by the following example:

import sys
import types

def imports():
    for name, val in globals().items():
        if isinstance(val, types.ModuleType):
            yield val.__name__

def worker():
    print('Worker modules:')
    print('\n'.join(imports()))

if __name__ == '__main__':
    import multiprocessing

    print('Main modules:')
    print('\n'.join(imports()))
    print()

    p = multiprocessing.Process(target=worker)
    p.start()
    p.join()

Output:

Main modules:
builtins
sys
types
multiprocessing

Worker modules:
sys
types

However, I don't think I can sell the rest of my team on wrapping the top-level script in if __name__ == '__main__' just to enable a small feature deep in the codebase. Still holding out hope that there's a way to do this without top-level changes.

python-3.x multiprocessing windows python
1 Answers

The docs you linked tells you:

Make sure that the main module can be safely imported by a new Python interpreter without causing unintended side effects (such a starting a new process).

...

Instead one should protect the “entry point” of the program by using if __name__ == '__main__': as follows:

...

You can also put import statements inside the if-block, then those import statements will only get executed when you run the __main__.py as a program but not when __main__.py is being imported.

<flame>Either that or switch to use a real OS that supports real fork()-ing</flame>

user_3094120
Related