Generate two independent dictionaries in parallel

Question

I want to build a dictionary of function evaluations in a parallel manner, but I am struggling to figure out how to do this efficiently.

Take the case of a randomly constructed matrix:

import functools
import multiprocessing
import numpy as np
import time

#generate random symmetric matrix
N = 500
b = np.random.random_integers(-2000,2000,size=(N,N))
b_symm = (b + b.T)/2

#identity matrix
ident = np.eye(N)

# define worker function:
def func(w, b_mat):

    if w !=0:
        L = np.linalg.inv(1j * w * ident - b_mat)

    else:
        L = np.linalg.pinv(-b_mat)

    return L

I now want to sample over many values of w, and construct a dictionary of outputs. This would be an embarrassingly parallel problem. I can do this by using a shared dictionary, using something like this:

def dict_builder(w, d):
    d[w] = func(w, b_symm)

manager = multiprocessing.Manager()

val_dict = manager.dict()


wrange = np.linspace(-10,10,200)

processors=2
pool = multiprocessing.Pool(processors)

st = time.time()
data = pool.map(functools.partial(dict_builder, d= val_dict), wrange,2)
pool.close()
pool.join()
en = time.time()
print("parallel test took ",en - st," seconds.")

but this seems more complicated than necessary since I am only evaluating the function at unique points, and comes with the overhead of having a shared memory object.

What I would like to is split wrange into n chunks, where n is the number of processors, build n dictionaries independently, then combine them into a single dictionary. So two questions: 1) Would this be computationally advantageous? 2) What is the best way to implement this using the multiprocessing module?

Generate two independent dictionaries in parallel

0 Answers