I'm pretty new to multiprocessing in python. Currently I have a function that I want to parallelize. This function will have the same input variables for all workers. One of the variables is a large list of objects, the other is a dictionary. In short I'm comparing objects in the list with items in the dictionary, and using a shared variable to keep track of what position of the list each worker is on. I've though of 2 ways of running this:
#option 1
def init(ind_arg,arg,arg2)
global index
global var_a
global var_b
index = ind_arg
var_a = arg
var_b = arg2
def worker(arg)
...code here...
def main()
index = mp.Value('i', 0)
thread_num = 8
var_a = ['']*16
var_b = ['']*8
pool = mp.Pool(thread_num, initializer = init, initargs = (index,var_a,var_b))
result_list = pool.map(worker, [''] * thread_num)
basically set global variables for each worker. I'm not sure if this works the way I think it works
#option 2
def init(ind_arg)
global index
index = ind_arg
def worker(c, a, b)
...code here using variable a and b...
def main()
index = mp.Value('i', 0)
thread_num = 8
var_a = ['']*16
var_b = ['']*8
pool = mp.Pool(thread_num, initializer = init, initargs = (index, ))
prod = partial(worker, a=var_a, b=var_b)
result_list = pool.map(prod, [''] * thread_num)
this one is straightforward just pass the variables using fuctools partial.
I'm sure there are other ways of doing this, if you know of a better one that would be cool to know. I'm also on windows so I cant use fork.
The data sets are relatively large so I would like to optimize for speed and/or memory usage. What option should I choose, or maybe something else, is best for performance?