Passing a Queue with concurrent.futures regardless of executor type

Viewed 926

Working up from threads to processes, I have switched to concurrent.futures, and would like to gain/retain flexibility in switching between a ThreadPoolExecutor and a ProcessPoolExecutor for various scenarios. However, despite the promise of a unified facade, I am having a hard time passing multiprocessing Queue objects as arguments on the futures.submit() when I switch to using a ProcessPoolExecutor:

import multiprocessing as mp
import concurrent.futures

def foo(q):
    q.put('hello')

if __name__ == '__main__':

    executor = concurrent.futures.ProcessPoolExecutor()
    q = mp.Queue()
    p = executor.submit(foo, q)
    p.result()
    print(q.get())

bumps into the following exception coming from multiprocessing's code:

RuntimeError: Queue objects should only be shared between processes through inheritance

which I believe means it doesn't like receiving the queue as an argument, but rather expects to (not in any OOP sense) "inherit it" on the multiprocessing fork rather than getting it as an argument.

The twist is that with bare-bones multiprocessing, meaning when not using it through the facade which concurrent.futures is ― there seems to be no such limitation, as the following code seamlessly works:

import multiprocessing as mp

def foo(q):
    q.put('hello')

if __name__ == '__main__':
    q = mp.Queue()
    p = mp.Process(target=foo, args=(q,))
    p.start()
    p.join()
    print(q.get())

I wonder what am I missing about this ― how can I make the ProcessPoolExecutor accept the queue as an argument when using concurrent.futures the same as it does when using the ThreadPoolExecutor or multiprocessing very directly like shown right above?

0 Answers
Related