MultiProcessing with ProcessPoolExcecutor gives un expected results

user_13757444 0
Viewed 52

Here i my code:

from concurrent.futures import ProcessPoolExecutor


class MyProcessPoolExecutor(ProcessPoolExecutor):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._running_workers = 0

    def submit(self, *args, **kwargs):
        future = super().submit(*args, **kwargs)
        self._running_workers += 1
        future.add_done_callback(self._worker_is_done)
        return future

    def _worker_is_done(self, future):
        self._running_workers -= 1

    @property
    def pool_usage(self):
        return self._running_workers


my_list = [1, 2, 3, 4, 5]
i = 0


def foo(x):
    global i
    i += 1
    if i <= 10:
        print(x)
        my_list.append(x)


if __name__ == "__main__":
    with MyProcessPoolExecutor() as executor:
        finished = False
        while not finished:
            while my_list:
                executor.submit(foo, my_list.pop(0))
            finished = (executor.pool_usage == 0 and not my_list)

Basicly what I try to achieve is to get the list "my_list" to be printes twice on multiple processes. Problem is my code stops after jut one "iteration" - i tried to add new while statement as shown above but still getting just "1,2,3,4,5" not "1,2,3,4,5,1,2,3,4,5" as I'd expect. Has anyone encountered this problem and can anyone help me?

while-loop list multiprocessing python
1 Answers

You are dealing with two different my_lists.

Add this to your program

def foo(x):
    print('id of my_list in launched process:', id(my_list))
    global i
    ...

if __name__ == '__main__':
    print('id of my_list in main process:', id(my_list))
    with MyProcessPoolExecutor() as executor:
        ...

and run it to see what's happening.

So you don't append anything to my_list in the memory space of the "coordinating" process and finished is True after 5 steps.

user_143112630
Related