Why thread.locks slowing down multiprocess pool.map function in python?

Viewed 17

ani have this function that calculate the total dot product for an array, but using pool map function in python slow it down instead of speed it up:

def parallel_execution(x,y):
  data_tuple=list(zip(x,y))
  with Pool(NUMCORES) as p:
    results_dot=p.starmap(my.dot,data_tuple)
    
  return sum(results_dot)

here is the dot function: `

def dot(a,b):
            r=a*b
            return r

here is the caller:

NUMDATA=10000000
data_X=np.random.rand(NUMDATA)
data_Y=np.random.rand(NUMDATA)

results_parallel=parallel_execution(data_X,data_Y)

profiling the function it says that the most of the time the function is spending is in thread locking how can i avoid this problem and what is this thread.lock method??

  ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   75   73.793    0.984   73.793    0.984 {method 'acquire' of '_thread.lock' objects}
0 Answers
Related