What actually happens when setting parallel=True in @njit numba?

user_12248220 0
Viewed 55

Could someone please explain roughly what happens when one runs a @njit-ted python function which contains a nested for loop (each iteration from each of the loops is independent of the others) and sets parallel=True and puts prange instead of range?

@njit(parallel=True)
def f():
    C = np.empty((80, 20, 18), dtype=np.complex128)
    for i in prange(80):
        for j in prange(20):
            for k in range(18):
            C[i, j, k] = do_smth(i, j, k) # where do_smth(i, j, k) is @njit-ted and will further call other functions

Similarly, what happens when using prange only for the outermost loop? (i.e. letting for j in range(20): ... )

I understand what a thread is and I put NUMBA_NUM_THREADS (the environmental variable) to be the number of cores of the processor.

I did some profiling using the timeit module and it seems that the parallel=True keyword only slows the execution of the f() function when the .py script is called on a machine with 20 cores (by a considerable amount (even 4 times slower)).

f() above further calls more functions (first one being do_smth()) also having their structure resembling the f()'s (nested for loops which, at each of their iterations, call other @njit-ted functions) structure. I checked them as above. Is my approach good? I.e. to profile them timeit and changing the keywords params inside their @njit decorator (I played with parallel, fastmath and nogil) and creating a table in which I note the execution times. My aim was to find the best execution time from the results I obtain.

numba parallel-processing optimization python performance
1 Answers

Could someone please explain roughly what happens when one runs a @njit-ted python function which contains a nested for loop and sets parallel=True and puts prange instead of range?

This is explained in the documentation, but basically, when parallel=True is set, prange split the loop iteration in blocks so they are executed in multiple threads. The exact scheduling is dependent of the underlying parallel runtime (eg. TBB, OpenMP, etc.). The loops is analyzed by Numba so to know whether a reduction is needed or not (not all patterns are allowed). It can also fuse parallel loops if needed (though it does not work on my machine with Numba 0.55.2, even on trivial reduction loops: only the outer loop is parallelized). Note that its takes time to create threads and the bigger the number of core, the slower it is. This is why multi-threaded computations should last for a relatively long time so for multiple threads to be useful.

Similarly, what happens when using prange only for the outermost loop? (i.e. letting for j in range(20): ... )

In theory, it is generally better to specify more parallelism. In practice, it is not always useful and sometimes even detrimental because the runtime can use inefficient methods (loop fusion can cause slow modulus to be used with some OpenMP backends).

If you use it only on the outer i-based loop, then only this loop is parallelized (using all the cores by default, so 4 iterations per loop if a static schedule is selected by the backend).

I did some profiling using the timeit module and it seems that the parallel=True keyword only slows the execution of the f() function when the .py script is called on a machine with 20 cores (by a considerable amount (even 4 times slower)).

Parallel programming is not easy. At least, far more than most people think. This is why researcher teams worked on it for decades and it is still an active field of research.

There are many effects that can be responsible for this, including:

  • Allocator contention (very frequent)
  • Undefined behavior in the code (frequent): typically a race condition (example)
  • False-sharing (quite frequent)
  • NUMA effects (eg. access to remote pages)
  • Other resource saturation (eg. memory) though it generally make the code barely scale and do not cause a slowdown (unless there is a contention)
  • A bug in Numba (quite rare)

Also note that the first call cause the function to be compiled so it is slower (and parallel codes are even slower to compile).

user_129395570
Related