How to deal with python3 multiprocessing in __main__.py

user_7156008 0
Viewed 255

Il posed question, I did not understand the ture cause of the issue (it seems to have been related to my usage of flask in one of the subprocesses).

PLEASE IGNORE THIS (can't delete due to bounty)

Essentially, I have to start some Processes and or a pool when running a python library as a module.

However, since __name__ == '__main__' is always true in __main__.py this proves to be an issue (see multiprocessing docs: https://docs.python.org/3/library/multiprocessing.html)

I've attempted multiple solutions ranging from: pytgquabr.com:8182/58288945/using-multiprocessing-with-runpy to a file-based mutext to only allow the contents of main to run once but multiprocessing still behaves strangely (e.g. Processes die almost as soon as they start with no error logs).

Any idea of what the "proper" way of going about this is ?

python-multithreading python-module python-3.x multiprocessing
1 Answers

Guarding the __main__ module is only needed if an object defined inside __main__ is used in another process. Looking up this definition is what causes the execution of __main__ in the subprocess.

When using a __main__.py, restrict all definitions used with multiprocessing to other modules. __main__.py should only import and use these.

# my_package/some_module.py
def module_print(*args, **kwargs):
    """Function defined in some module - fine for use inside multiprocess"""
    print(*args, **kwargs)

# my_package/__main__.py
import multiprocessing                 # imports are allowed
from .some_module import module_print

def do_multiprocess():
    """Function defined in __main__ module - fine for use wrapping multiprocess"""
    with multiprocessing.Pool(processes=12) as pool:
        pool.map(module_print, range(20))  # multiprocessing external function is allowed

do_multiprocess()  # directly calling __main__ function is allowed
user_53499160
Related