Multiprocessing with lower python version executable

user_19043397 0
Viewed 65

I need to execute a python function with python3.8 and I'm running my main program on python3.9.

My main program looks like this:

import multiprocessing as mp


def run(function, executable):
    mp.set_start_method("spawn", force=True)
    mp.set_executable(executable)
    p = mp.Process(target=function)
    p.start()


if __name__ == "__main__":
    from module import foo
    run(
        function=foo.bar,
        executable="/bin/python3.8",
    )

The bar function from the foo module is like this:

import pathlib
def bar():
    """Do some python3.8 here"""

When importing packages in the code executed by python3.8 (checked by printing sys.executable), it seems to import the packages from python3.9. In my case I have this issue with the pathlib package, but the same happened with numpy:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/usr/lib/python3.8/multiprocessing/spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
  File "/gitrepos/mp-issue/module/foo.py", line 1, in <module>
    import pathlib
  File "/usr/lib/python3.9/pathlib.py", line 13, in <module>
    from urllib.parse import quote_from_bytes as urlquote_from_bytes
  File "/usr/lib/python3.9/urllib/parse.py", line 147, in <module>
    class _NetlocResultMixinBase(object):
  File "/usr/lib/python3.9/urllib/parse.py", line 183, in _NetlocResultMixinBase
    __class_getitem__ = classmethod(types.GenericAlias)
AttributeError: module 'types' has no attribute 'GenericAlias'

I guess that I will have this issue with all the 3.9 packages not compatible with 3.8. I tried moving the import in the bar function but the error still happened.

Am I misusing multiprocessing, or should I do this in an other way ?

multiprocessing python
2 Answers

I would first ensure that module foo was available to Python 3.8, which should be the case if it is currently a module being loaded from a local directory (i.e. not installed with pip). Otherwise, you will need to install foo in Python 3.8. If loading foo currently depends on environment variable PYTHONPATH being set, then you will need to pass the current environment by setting variable MUST_PASS_PYTHONPATH to True:

if __name__ == "__main__":
    import subprocess

    MUST_PASS_PYTHONPATH = False

    # pass current environment in case PYTHONPATH is required
    d = dict(os.environ) if MUST_PASS_PYTHONPATH else None

    subprocess.run(['/bin/python3.8', '-c', 'from module import foo; foo()'], env=d)
user_28237190

This happens because the sys.path is not altered for the child processes. You need to add the paths for the libraries (built-in or otherwise) inside where python3.8 is installed to the sys.path for the child processes. You could make a decorator which does that automatically whenever the function is called, but you will need to include every import you do inside every function you are planning to call with multiprocessing. Additionally, decorators are factory functions which cannot be pickled using multiprocessing in the standard library (you will have to use multiprocess from outside the standard library to do that that).

A better solution would perhaps be to add the paths whenever the module foo is imported. This would mean the sys.path will be altered for your main process as well, but there is a workaround which includes letting your module know which executable you'll be calling it from. General solution for this would likely be error-prone and rather tricky to implement (for example, you could have virtual environments setup which add their own sys.path). Having said that, here is something you can do:

Inside foo.py

import os
import sys

def prepare():
    dir_name = os.path.dirname(sys.executable)
    to_add = [None, 'lib', 'DLLs', f'python{sys.version_info.major}{sys.version_info.minor}.zip']

    for name in to_add:
        if name is None:
            sys.path.insert(0, dir_name)
            continue
        sys.path.insert(0, os.path.join(dir_name, name))


if sys.executable == r"path\to\python.exe":
    prepare()


# Your imports come here, make sure to import os and sys again!
import pathlib, sys, os

def bar():
    print('ok')

Make sure to edit path/to/python.exe to the actual executable file location. This works because unlike sys.path, sys.exectuable is altered by multiprocessing when using the set_executable_location function to spawn children.

The general idea you should take away is that you need to change sys.path of child processes so that they point to where the libraries must be imported from if you are changing the python version of the interpreter. Multiprocessing will not do that for you.

user_163107410
Related