How to make use of a multiprocessing manager within a class

user_4391249 0
Viewed 1047

To start with, here is some code that works

from multiprocessing import Pool, Manager
import random

manager = Manager()
dct = manager.dict()

def do_thing(n):
    for i in range(10_000_000):
        i += 1
    dct[n] = random.randint(0, 9)

with Pool(2) as pool:
    pool.map(do_thing, range(10))

Now if I try to make a class out of this:

from multiprocessing import Pool, Manager
import random


class SomeClass:
    def __init__(self):
        self.manager = Manager()
        self.dct = self.manager.dict()

    def __call__(self):
        with Pool(2) as pool:
            pool.map(self.do_thing, range(10))

    def do_thing(self, n):
        for i in range(10_000_000):
            i += 1
        self.dct[n] = random.randint(0, 9)


if __name__ == '__main__':
    inst = SomeClass()
    inst()

I run into: TypeError: Pickling an AuthenticationString object is disallowed for security reasons. Now from here, I get the hint that Python is trying to pickle the Manager which as I understand has its own dedicated process, and processes can't be pickled because they contain an AuthenticationString.

I don't know enough about how forking works (I'm on Linux, so I understand this is the default method for starting new processes) to understand exactly why the Manager instance needs to be pickled.

So here are my questions:

  1. Why is this happening?
  2. How can I use a Manager when doing multiprocessing within a class? PS: I want to be able to import SomeClass from this module.
  3. Is what I'm asking for unreasonable or unconventional?

PS: I know I can do this exact snippet without the Manager by exploiting the fact that pool.map will return things in order, so something like this: res = pool.map(self.do_thing, range(10)) then dct = {k: v for k, v in zip(range(10), res)}. But that's besides the point of the question.

multiprocessing python
1 Answers

To answer your questions:

Q1 - Why is this happening?

Each worker process created by the Pool.map() needs to execute the instance method self.do_thing(). In order to do that Python pickles the instance and passes it to the subprocess (which unpickles it). If each instance has a Manager it will be a problem because they're not pickleable. Part of the unpickling process involves importing the module that defines the class and restoring the instance's attributes (which were also pickled).

Q2 - How to fix it

You can avoid the problem by having the class create its own class-level Manager (shared by all instances of the class). Here the __init__() method creates the manager class attribute the first time an instance is created and from that point on, further instances will reuse this — it's sometimes called "lazy initialization"

from multiprocessing import Pool, Manager
import random


class SomeClass:
    def __init__(self):
        # Lazy creation of class attribute.
        try:
            manager = getattr(type(self), 'manager')
        except AttributeError:
            manager = type(self).manager = Manager()
        self.dct = manager.dict()

    def __call__(self):
        with Pool(2) as pool:
            pool.map(self.do_thing, range(10))
        print('done')

    def do_thing(self, n):
        for i in range(10_000_000):
            i += 1
        self.dct[n] = random.randint(0, 9)


if __name__ == '__main__':
    inst = SomeClass()
    inst()

Q3 - Is this a reasonable thing to do?

In my opinion, yes.

user_3552300
Related