multiprocessing in dynamically imported module

Viewed 196

I have two classes A and B. A should import B dynamically (so I can change B during runtime). I also want to do some multiprocessing in B.

This is my directory structure:

src\
  __init__.py
  a\
    __init__.py
    a.py
  b\
    __init__.py
    b.py

This is a.py:

import os, importlib.util

if __name__ == '__main__':
  module_path = 'src\\b\\b.py'
  module_name = 'b'
  class_name = 'B'

  # dynamically importing the module
  spec = importlib.util.spec_from_file_location(module_name, module_path)
  module = importlib.util.module_from_spec(spec)
  spec.loader.exec_module(module)

  # dynamically instantiating the class
  b_class = getattr(module, class_name)
  b = b_class()

  # start multiprocessing
  b.run()

This is b.py:

import multiprocessing as mp

class B():
  def f(self):
    print('success')

  def run(self):
    print('run')
    p = mp.Process(target=self.f)
    p.start()
    p.join()

if __name__ == '__main__':
  b = B()
  b.run()

Running b.py works fine:

> python src\b\b.py
run
success

Running a.py raises an exception:

> python src\a\a.py
run
Traceback (most recent call last):
  File "src\a\a.py", line 18, in <module>
    b.run()
  File "src\b\b.py", line 10, in run
    p.start()
  File "C:\Program Files\Python38\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Program Files\Python38\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Program Files\Python38\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "C:\Program Files\Python38\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Program Files\Python38\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class 'b.B'>: import of module 'b' failed

C:\Users\micha\Projects\Python\PickleQuestion>Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\Python38\lib\multiprocessing\spawn.py", line 102, in spawn_main
    source_process = _winapi.OpenProcess(
OSError: [WinError 87] Falscher Parameter

Why? Is there another way to import B, while keeping the flexibility of not hardcoding it (from b import B)? I want b.py to be in any package, even outside of src.

Python version: 3.8.6

0 Answers
Related