Any way to share a (class) instance argument while multiprocessing in python

Yanghoon 0
Viewed 38

I made an simple example to describe the problem I am facing.

code example1:

from multiprocessing import Pool
from functools import partial

metric # a class instance which includes several pandas dataframe as class variable. This instance is defined globally


variables # a list of size 2000k

def get_reward(variable_, metric_, *args):
    ...
    reward = metric_.some_method(variable_)
    return reward

func_ = partial(get_reward, metric_=metric)

mp_workers = Pool(64)
map_ = mp_workers.map(func_, variables)

code example2:

from multiprocessing import Pool
from functools import partial

metric # a class instance which includes several pandas dataframe as class variable. This instance is defined globally


variables # a list of size 2000k

def get_reward(variable_, *args):
    ...
    reward = metric.some_method(variable_)
    return reward


mp_workers = Pool(64)
map_ = mp_workers.map(get_reward, variables)

In a nutshell, the difference between the two code examples is that: the 'get_reward' function in the second code doesn't take the class instance (metric) as argument and just use it as the global instance, while the 'get_reward' function in the first code take the class instance as an argument.

I found that the first code work very slowly while the second code finely works. The reason may be that the first code copies the big class instance to every worker and it takes quite a long time.

In my real code, I can only have the structure like the first code example, in which the 'get_reward' function should take the class instance as an argument.

Any way to accelerate the code example 1?

Thanks.

python-multiprocessing multiprocessing python
1 Answers

Each Python Process lives in its own address space. It cannot directly share data with another Process; instead, Pipes and Queues must be used to transfer bytes from one Process to another. There is no way to directly "share" an object, such as a dataframe. Standard library functions use the Python pickle mechanism to encode an object to a byte stream, send it to another Process, and reconstitute it as an object on the other side.

However, in the case of your metric object, your example as written will pickle/transfer/unpickle it 2000k times, once for every item in variables. That's what the map function does. Obviously this is terribly inefficient.

One way to improve this is to initialize each Process once, with its own copy of the dataframe. Individual data items are placed on a Queue and removed by the Processes, as soon as they are done with the previous item. No extra pickling takes place. You lose the convenience of the map function, but that's life.

It is probably best to create only as many Processes as you have CPU cores, since you can't do better than keep all the cores busy.

Here is a stupid little program to create two Processes, initialize them with a data object and a queue, and start their execution. The time stamps show that the two Processes are actually running in parallel.

from multiprocessing import Process, SimpleQueue
import time
from datetime import datetime

class Process1(Process):
    def __init__(self, queue, an_obj):
        self.queue = queue
        self.an_obj = an_obj
        super().__init__()
        print("Process1", an_obj, self.pid)
        
    def run(self):
        while not self.queue.empty():
            print(self.queue.get(), datetime.now(), self.an_obj)
            time.sleep(1.0)
            
def main():
    q = SimpleQueue()
    p1 = Process1(q, "I wish I were a dataframe")
    p2 = Process1(q, "I wish I were a dataframe too")
    for n in range(20):
        q.put(n)
    p1.start()
    p2.start()
    p1.join()
    p2.join()

Output:

Process1 I wish I were a dataframe None
Process1 I wish I were a dataframe too None
0 2022-09-07 18:52:14.988650 I wish I were a dataframe
1 2022-09-07 18:52:15.004269 I wish I were a dataframe too
2 2022-09-07 18:52:15.990152 I wish I were a dataframe
3 2022-09-07 18:52:16.005771 I wish I were a dataframe too
4 2022-09-07 18:52:17.005312 I wish I were a dataframe
5 2022-09-07 18:52:17.020999 I wish I were a dataframe too
6 2022-09-07 18:52:18.005842 I wish I were a dataframe
7 2022-09-07 18:52:18.021450 I wish I were a dataframe too
8 2022-09-07 18:52:19.006774 I wish I were a dataframe
9 2022-09-07 18:52:19.022367 I wish I were a dataframe too
10 2022-09-07 18:52:20.007715 I wish I were a dataframe
11 2022-09-07 18:52:20.023279 I wish I were a dataframe too
12 2022-09-07 18:52:21.022944 I wish I were a dataframe
13 2022-09-07 18:52:21.038544 I wish I were a dataframe too
14 2022-09-07 18:52:22.023039 I wish I were a dataframe
15 2022-09-07 18:52:22.038665 I wish I were a dataframe too
16 2022-09-07 18:52:23.024069 I wish I were a dataframe
17 2022-09-07 18:52:23.039694 I wish I were a dataframe too
18 2022-09-07 18:52:24.025000 I wish I were a dataframe
19 2022-09-07 18:52:24.040682 I wish I were a dataframe too
Paul Cornelius0
Related