how can i improve the speed of this function:
Now the speed is around 2 seconds, but it takes a lot to zip the arrays and split it in 32 sub-arrays(around 10/15 secs)
if i do this without multiprocessing it takes less seconds to create the array to process(around 4-6 sec) but 6 seconds for executing the function
is it possible to do it faster in parallel?
def parallel_execution_improved(x):
with Pool(NUMCORES) as p:
results=p.map(my.newdot,x)
return sum(results)
NUMDATA=10000000
data_X=np.random.rand(NUMDATA)
data_Y=np.random.rand(NUMDATA)
start_time0 = time.time()
data_tuple=list(zip(data_X,data_Y))
newarr = np.array_split(data_tuple, 32)
print("--- %s creation time ---" % (time.time() - start_time0))
start_time = time.time()
results_parallel=parallel_execution_improved(newarr)
print("--- %s function time ---" % (time.time() - start_time))
print(results_parallel)
and the newdot function:
def newdot(a):
r=0
for x in a:
r+=(x[0]*x[1])
return r