multiprocessing_generator modules triggers a permission error

user_5736515 0
Viewed 543

I'm found the module multiprocessing_generator. I tried that module with the code below :

from multiprocessing_generator import ParallelGenerator

def my_generator():
    yield (x*x for x in range(200))

with ParallelGenerator(my_generator(), max_lookahead=100) as g:
    for elem in g:
        print(elem)

Here is the error I got (I ran my code in the console, the python file is in my desktop) :

C:\Users\crd\Desktop>python test.py Traceback (most recent call last): File "test.py", line 69, in with ParallelGenerator(my_generator(), max_lookahead=100) as g: File "C:\Users\crd\AppData\Local\Programs\Python\Python37-32\lib\site-packages\multiprocessing_generator__init__.py", line 62, in enter self.process.start() File "C:\Users\crd\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\process.py", line 112, in start self._popen = self._Popen(self) File "C:\Users\crd\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\context.py", line 223, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "C:\Users\crd\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj) File "C:\Users\crd\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\popen_spawn_win32.py", line 65, in init reduction.dump(process_obj, to_child) File "C:\Users\crd\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) AttributeError: Can't pickle local object 'ParallelGenerator.init..wrapped'

C:\Users\crd\Desktop>Traceback (most recent call last): File "", line 1, in File "C:\Users\crd\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\spawn.py", line 99, in spawn_main new_handle = reduction.steal_handle(parent_pid, pipe_handle) File "C:\Users\crd\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\reduction.py", line 87, in steal_handle _winapi.DUPLICATE_SAME_ACCESS | _winapi.DUPLICATE_CLOSE_SOURCE) PermissionError: [WinError 5] Accès refusé

I cleaned up the code from the multiprocessing_generator module (I deleted every try/exeption, I converted the context-manager into a basic function). But with the code below I got the same error :

from queue import Empty

from multiprocessing import Process, Queue

def ParallelGeneratorLight():
    queue = Queue()

    def wrapped():
        for item in (x*x for x in range(200)):
            queue.put(item)

    process = Process(target=wrapped)

    process.start()
    queue.get()

print(ParallelGeneratorLight())

What is wrong with that ?

generator python-3.x multiprocessing python
1 Answers

multiprocessing_generator relies on the multiprocessing module using the 'fork' start method, because it assumes that the wrapped nested function (defined in ParallelGenerator's __init__) can be passed as the target of a multiprocessing.Process object. On a forking based platform, this is fine; the child process inherits the complete state of the parent (with a few small exceptions like threads), so it has equal access to the wrapped nested function (it inherited an exact copy after all).

Problem is, on Windows, the only available start method is 'spawn', which requires the target (and all arguments) to be pickleable (it pickles them, sends them to the child over IPC, and reconstructs them there), and nested functions are never pickleable (pickleing a function involves pickling its qualified name to be imported and used on the other side, and the qualified name of a nested function involves unimportable components, in this case, pickleing ParallelGenerator.__init__.<locals>.wrapped fails because <locals> is clearly not an importable name).

Basically, multiprocessing_generator only works on UNIX-like systems, and only if you're using the default start method ('fork'); if you called set_start_method with some other value ('forkserver' or 'spawn'), multiprocessing_generator can't be made to work.

While this is a bug, it's not a particularly critical bug in most cases; there is very little benefit to the module if the generator must have its values pickled, because most such generators either aren't pickleable themselves (e.g. most file-like objects) or pickleing them involves running them to completion (in which case you've lost all your parallelism).

Sorry, but the simple answer here is: Don't use multiprocessing_generator on Windows.

That said, if your generator is I/O-bound, you might be able to benefit from the module by importing it, then immediately monkey-patching it to replace all multiprocessing components it relies on with their equivalent multiprocessing.dummy names (which are backed by threads, and don't rely on pickling), e.g.

from multiprocessing_generator import ParallelGenerator
import multiprocessing.dummy, multiprocessing_generator

# Monkey-patch to use threads
multiprocessing_generator.Process = multiprocessing.dummy.Process
multiprocessing_generator.Queue = multiprocessing.dummy.Queue

To be clear, I have not tested this; no warranties are expressed or implied as to whether this will work. It will also be completely pointless if the generator is CPU-bound, at least on the CPython reference interpreter, since CPU-bound generators will hold the GIL while running, preventing the main thread from performing work, so you may as well have iterated directly.

user_3646960
Related