Class serialization when member function is invoked by JobLib

Viewed 107

I would appreciate any help or suggestion from the community regarding the following code example, see below.

In this example I invoke class member function from another member function in parallel. It turns out, when a member function self.print_id() is called by JobLib framework, then

  1. the entire class is cloned (without calling the constructor), serialized and passed to a process.
  2. a new unique ID is assigned to each copy of the class.

Questions:

  1. Is this a correct construction?
  2. Is there any apparent side effect that might happen?
  3. What if the "class A" is inherited, can I expect the correct behaviour of self.print_id() member of a derived class?

Thank you in advance.

from joblib import Parallel, delayed


class A:
    def __init__(self):
        self.state = -1
        print(f"constructor is called, id: {id(self)}", flush=True)

    def print_id(self, index):
        self.state += 1
        print(f"state: {self.state}, id: {id(self)}, index: {index}")

    def parallel(self):
        Parallel(n_jobs=8, prefer="processes")(
            delayed(self.print_id)(i) for i in range(8))


a = A()
a.print_id(777)
a.parallel()

constructor is called, id: 140331251437424
state: 0, id: 140331251437424, index: 777
state: 1, id: 140226941702448, index: 0
state: 1, id: 140520215830832, index: 1
state: 1, id: 140520209383920, index: 2
state: 1, id: 140520215850624, index: 3
state: 1, id: 140520209384352, index: 4
state: 1, id: 140520215621200, index: 5
state: 1, id: 140520209384832, index: 6
state: 1, id: 140520215621296, index: 7
0 Answers
Related