Python multiprocessing.Pool.map behavior when list is longer than number of processes

user_9779026 0
Viewed 1466

When submitting a list of tasks that is longer than the number of processes, how are the processes assigned to these tasks?

from multiprocessing import Pool

def f(i):
    print(i)
    return i

with Pool(2) as pool:
    print(pool.map(f, [1, 2, 3, 4, 5]))

I'm running a more complex function and the execution doesn't seem to be in order (FIFO).

python-multiprocessing multiprocessing python
2 Answers

The Pool class represents a pool of worker processes. It spawns new process as soon as an one on the existing process finishes. To understand better, we set a chunksize=1, consider the code,

from multiprocessing import Pool
from time import sleep


def f(x):
    print(f"Task {x} enter")
    sleep(5)
    print(f"Task {x} exit")
    return x * x


if __name__ == '__main__':
    with Pool(2) as pool:
        print(pool.map(f, range(10), chunksize=1))

So the order of execution will be,

Task 0 enter
Task 1 enter
Task 0 exit
Task 2 enter
Task 1 exit
Task 3 enter
Task 2 exit
Task 4 enter
Task 3 exit
Task 5 enter
Task 4 exit
Task 6 enter
Task 5 exit
Task 7 enter
Task 6 exit
Task 8 enter
Task 7 exit
Task 9 enter
Task 8 exit
Task 9 exit
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
user_128331660

Here's some sample code:

from multiprocessing import Pool
from time import sleep


def f(x):
    print(x)
    sleep(0.1)
    return x * x


if __name__ == '__main__':
    with Pool(2) as pool:
        print(pool.map(f, range(100)))

Which prints out:

0
13
1
14
2
15
3
16
4
...

If we look into the relevant source code in multiprocessing:

    def _map_async(self, func, iterable, mapper, chunksize=None, callback=None,
            error_callback=None):
        '''
        Helper function to implement map, starmap and their async counterparts.
        '''
        self._check_running()
        if not hasattr(iterable, '__len__'):
            iterable = list(iterable)

        if chunksize is None:
            chunksize, extra = divmod(len(iterable), len(self._pool) * 4)
            if extra:
                chunksize += 1
        if len(iterable) == 0:
            chunksize = 0

        task_batches = Pool._get_tasks(func, iterable, chunksize)

Here we have len(iterable) == 100, len(self._pool) * 4 == 8, so chunksize, extra = 12, 4 which leads to chunksize = 13, hence the output shows the tasks being split into batches of 13.

user_24827440
Related