Multiprocessing taking longer than single (normal) processing

user_4466239 0
Viewed 143

How come the multi function that uses a multiprocessing pool to segment and process data on multiple "processes" is slower (8 seconds) than just calling the map function (6 seconds)?

from multiprocessing import Pool
import timeit


def timer(function):
  def new_function():
    start_time = timeit.default_timer()
    function()
    elapsed = timeit.default_timer() - start_time
    print('Function "{name}" took {time} seconds to complete.'.format(name=function.__name__, time=elapsed))
  return new_function


def cube(n):
    return n*n*n

nums = range(20000000)
if __name__ == '__main__':
    @timer
    def multi():
        
        pool = Pool()
        res = pool.map(cube,nums)
        pool.close()
        pool.join()

    @timer
    def test():
        a = map(cube,nums)
            
    multi()
    test()
python-3.x multithreading multiprocessing python
1 Answers

Because all the dispatching logic behind pool.map creates an overhead.

Multiprocessing always create overhead of some sort, which heavily depends on its underlying implementation.

You are running a lot of very simple tasks here, hence the overhead caused by the threading logic is not compensated by the gain of parallel execution. Try to do the same test with a lesser number of more cpu-intensive task, you should see different results.

Exemple

See this modified test. Here, we have a silly cubes function that computes n^3 1000 times.

from multiprocessing import Pool
import timeit


def timer(function):
    def new_function():
        start_time = timeit.default_timer()
        function()
        elapsed = timeit.default_timer() - start_time
        print('Function "{name}" took {time} seconds to complete.'.format(name=function.__name__, time=elapsed))
    return new_function


def cubes(n):
    for _ in range(999):
        n * n * n
    return n * n * n

nums = range(20000)
if __name__ == '__main__':
    @timer
    def multi():
        pool = Pool()
        res = pool.map(cubes, nums)
        pool.close()
        pool.join()

    @timer
    def test():
        # On Python 3, simply calling map() returns an iterator
        # tuple() collects its values for timing
        a = tuple(map(cubes, nums))
            
    multi()
    test()

We now see multiprocessing is improving our timing:

Function "multi" took 0.6272498000000001 seconds to complete.
Function "test" took 2.130454 seconds to complete.
user_70212230
Related