Multiprocessing with `Pool` throws error on M1 Macbook

user_3986879 0
Viewed 2228

On my Macbook Pro (Intel, 2020) I can successfully use multiprocessing.Pool like:

from multiprocessing import Pool

p = Pool(8)
results = p.map(worker_function, list_of_inputs)
p.close()

However, if I run the same code on my Macbook Air (M1, 2020), I get a strange error repeated again and again (snippet below):

RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
apple-m1 python-multiprocessing multiprocessing python
1 Answers

To solve this I first read this blog post. The author explains the different ways that Python can start new threads, e.g. by fork-ing (basically copying the existing interpreter and most of its memory), spawn-ing new interpreters, and so on.

In the official documentation, it appears that the default start method on OSX is fork. However, I noticed, that on my Macbook (M1, 2020), if I run:

import multiprocessing

multiprocessing.get_start_method()

I get "spawn".

So I managed to solve the problem by explicitly declaring that I want the "fork" start method when creating the pool.

from multiprocessing import get_context

p = get_context("fork").Pool(8)
results = p.map(worker_function, list_of_inputs)
p.close()

Update

Have you checked out ray? It's faster and has a much cleaner API for multiprocessing. The above would simply be:

import ray
results = ray.get([worker_funtion.remote(inp) for inp in list_of_inputs])
user_39868790
Related