How to use joblib parallelization with in-class methods that don't return anything

user_11069310 0
Viewed 782

I am currently trying to implement a parallel for loop using joblib in python 3.8.3.

In the for loop I want to apply a class method to an instance of one class while applying a method in another. This is a MWE that I did to try and see if my idea works, but it does not. Does anyone have any idea how to get this to work?

from joblib import Parallel, delayed

class A():
    def __init__(self):
        self.val = 0
    def add5(self):
        self.val += 5

class B():
    def __init__(self):
        self.obj = [A() for _ in range(10)]
    def apply(self):
        """ this is where I'm trying to use joblib:
        for a in self.obj:
            a.add5()"""

        def f(x):
            x.add5()
        Parallel(n_jobs=-1)(delayed(f)(x) for x in self.obj)
    def prnt(self):
        print([a.val for a in self.obj])

b = B()
b.prnt()  # returns [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
b.apply()
b.prnt()  # returns [0, 0, 0, 0, 0, 0, 0, 0, 0, 0] but
          # I expect [5, 5, 5, 5, 5, 5, 5, 5, 5, 5]

Some more context to my problem: I am using sci-kit learn to implement a boosting algorithm where I generate weak learners before applying the algorithm. The fitting and predicting is done in a for-loop and can take some time, so I want to add parallelization to try and speed up the progress. Basically, class A is a classifier and class B is my algorithm, where I want to fit all classifiers that I generated.

joblib scikit-learn parallel-processing multithreading python
1 Answers

From (source) one can read that:

The default backend of joblib will run each function call in isolated Python processes, therefore they cannot mutate a common Python object defined in the main program.

However if the parallel function really needs to rely on the shared memory semantics of threads, it should be made explicit with require='sharedmem', for instance:

So you have two options: 1) you add require='sharedmem' to your Parallel For:

Parallel(n_jobs=-1, require='sharedmem')(delayed(f)(x) for x in self.obj)

However, source points out that:

Keep in mind that relying a on the shared-memory semantics is probably suboptimal from a performance point of view as concurrent access to a shared Python object will suffer from lock contention.

In the 2) option, you will have to change two things in your code.

First change the f function from:

 def f(x):
      x.add5()

to return the object back.

def f(x):
    x.add5()
    return x

and in the Parallel loop, change from:

  Parallel(n_jobs=-1)(delayed(f)(x) for x in self.obj)

into:

 self.obj = Parallel(n_jobs=-1)(delayed(f)(x) for x in self.obj)

So that you can assign the self.obj to the list return by the parallel loop.

Final code:

from joblib import Parallel, delayed


class A:
    def __init__(self):
        self.val = 0

    def add5(self):
        self.val += 5


class B:
    def __init__(self):
        self.obj = [A() for _ in range(10)]

    def apply(self):
        """ this is where I'm trying to use joblib:
        for a in self.obj:
            a.add5()"""

        def f(x):
            x.add5()
            return x

        self.obj = Parallel(n_jobs=-1)(delayed(f)(x) for x in self.obj)

    def prnt(self):
        print([a.val for a in self.obj])


b = B()
b.prnt()  # returns [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
b.apply()
b.prnt()  # returns [0, 0, 0, 0, 0, 0, 0, 0, 0, 0] but
          # I expect [5, 5, 5, 5, 5, 5, 5, 5, 5, 5]
user_13668710
Related