Improve a multiprocess function in python

Viewed 39

how can i improve the speed of this function:

Now the speed is around 2 seconds, but it takes a lot to zip the arrays and split it in 32 sub-arrays(around 10/15 secs)

if i do this without multiprocessing it takes less seconds to create the array to process(around 4-6 sec) but 6 seconds for executing the function

is it possible to do it faster in parallel?

def parallel_execution_improved(x):

    with Pool(NUMCORES) as p:
            results=p.map(my.newdot,x)
            
    return sum(results)


NUMDATA=10000000
data_X=np.random.rand(NUMDATA)
data_Y=np.random.rand(NUMDATA)

start_time0 = time.time()
data_tuple=list(zip(data_X,data_Y))
newarr = np.array_split(data_tuple, 32)
print("--- %s creation time ---" % (time.time() - start_time0))

start_time = time.time()
results_parallel=parallel_execution_improved(newarr)
print("--- %s function time ---" % (time.time() - start_time))
print(results_parallel)

and the newdot function:

def newdot(a):
    r=0
    for x in a:
        
        r+=(x[0]*x[1])
    return r
0 Answers
Related