Python 3.8 multiprocessing: TypeError: cannot pickle 'weakref' object

user_6824949 0
Viewed 1307

We are trying to execute this piece of code using the multiprocessing module:

import multiprocessing as mp

ctx = mp.get_context("spawn")
(child, pipe) = ctx.Pipe(duplex=True)
job_process = ctx.Process(
    name="my_job",
    target=job_func,
    args=(
        child,
        server_info,
        manager,
        job_config,
        config_file,
    ),
)
job_process.start()

And we run into the below error in Python 3.8 that is not seen in Python 3.6:

 Traceback (most recent call last):
   File "path/to/venv/lib/python3.8/site-packages/path/to/file.py", line 137, in includeme 
     job_process.start()
   File "/usr/lib/python3.8/multiprocessing/process.py", line 121, in start
     self._popen = self._Popen(self)
   File "/usr/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
     return Popen(process_obj)
   File "/usr/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
     super().__init__(process_obj)
   File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
     self._launch(process_obj)
   File "/usr/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
     reduction.dump(process_obj, fp)
   File "/usr/lib/python3.8/multiprocessing/reduction.py", line 63, in dump
     ForkingPickler(file, protocol).dump(obj)
 TypeError: cannot pickle 'weakref' object

Is there something in the way the process is started or the arguments supplied that needs to be changed or checked? Any direction is appreciated!

python-3.x multiprocessing python
1 Answers

This problem is likely a side-effect of a bigger bug in your application. You must use import guards when using multiprocessing in 'spawn' mode, and failing to do so does some weird things. In particular:

  1. The fork-emulation done in spawn mode will try to serialize some of the main module's state for transfer to the child process to initialize the child similarly; multiprocessing.Process itself is not picklable as of Python 3.7 (though there's a patch pending to fix it), so you definitely don't want to have it anywhere it might get pickled.

  2. By not using the guards, the child process's import of the main module does all the things the main module does, including launching a child (which itself launches a child, and so on, ad infinitum).

Both problems should be solved by using the import guard and putting everything in a main function called within the guard:

import multiprocessing as mp

# Wrapping in function avoids implicit dependencies on mutable global state
def main():
    ctx = mp.get_context("spawn")
    (child, pipe) = ctx.Pipe(duplex=True)
    job_process = ctx.Process(
        name="my_job",
        target=job_func,
        args=(
            child,
            server_info,
            manager,
            job_config,
            config_file,
        ),
    )
    job_process.start()

# Guard ensures only the parent process tries to actually run main()
if __name__ == '__main__':
    main()
user_3646960
Related