Python passing same variables into a multiprocessing pool: global variables vs partial

Viewed 9

I'm pretty new to multiprocessing in python. Currently I have a function that I want to parallelize. This function will have the same input variables for all workers. One of the variables is a large list of objects, the other is a dictionary. In short I'm comparing objects in the list with items in the dictionary, and using a shared variable to keep track of what position of the list each worker is on. I've though of 2 ways of running this:

#option 1
def init(ind_arg,arg,arg2)
   global index 
   global var_a
   global var_b
   index = ind_arg
   var_a = arg
   var_b = arg2

def worker(arg)
   ...code here...

def main()
   index = mp.Value('i', 0)
   thread_num = 8
   var_a = ['']*16
   var_b = ['']*8
   pool = mp.Pool(thread_num, initializer = init, initargs = (index,var_a,var_b))
   result_list = pool.map(worker, [''] * thread_num)

basically set global variables for each worker. I'm not sure if this works the way I think it works

#option 2
def init(ind_arg)
   global index 
   index = ind_arg

def worker(c, a, b)
   ...code here using variable a and b...

def main()
   index = mp.Value('i', 0)
   thread_num = 8
   var_a = ['']*16
   var_b = ['']*8
   pool = mp.Pool(thread_num, initializer = init, initargs = (index, ))
   prod = partial(worker, a=var_a, b=var_b)
   result_list = pool.map(prod, [''] * thread_num)

this one is straightforward just pass the variables using fuctools partial.

I'm sure there are other ways of doing this, if you know of a better one that would be cool to know. I'm also on windows so I cant use fork.

The data sets are relatively large so I would like to optimize for speed and/or memory usage. What option should I choose, or maybe something else, is best for performance?

0 Answers
Related