Parallel compilation of several numba functions

user_13544828 0
Viewed 518

I am trying to use numba to compile several functions within my codebase to speed up execution. I have several thousand standalone functions that I am trying to compile with the jit decorator. While compiling my functions I notice that all the code is compiled within a single python process and only makes use of a single core within my machine. This results in compilation times often exceeding an hour. I am wondering if I can enable parallelization during the compilation step similar to make -j <number of processors>.

My functions are completely independent of each other and subsequent calls that utilize them are in a separate sub-module. Here is an example of the sort of functions that I would like to be able to compile in parallel:

import numpy as np
from numba import jit, float64
@jit(float64[:](float64[:]),nopython=True,cache=True,fastmath=True,parallel=True,nogil=True)
def fun_0(x):
  return np.power(x,0)

@jit(float64[:](float64[:]),nopython=True,cache=True,fastmath=True,parallel=True,nogil=True)
def fun_1(x):
  return np.power(x,1)

@jit(float64[:](float64[:]),nopython=True,cache=True,fastmath=True,parallel=True,nogil=True)
def fun_2(x):
  return np.power(x,2)

@jit(float64[:](float64[:]),nopython=True,cache=True,fastmath=True,parallel=True,nogil=True)
def fun_3(x):
  return np.power(x,3)

The code is only illustrative of my use case. Alternate ways to structure my functions/modules may also be an acceptable solution for me.

numba multicore parallel-processing compilation python
1 Answers

Is the goal to increase the runtime performance, observed to get lowered by the @decorator-"inlined" JIT compilation of several thousand standalone functions?

Solution :
may use "AoT Compilation: i.e. Compiling code ahead of time"

While Numba’s main use case is Just-in-Time compilation, it also provides a facility for Ahead-of-Time compilation (AOT).

Limitations :
1) AOT compilation only allows for regular functions, not ufuncs.
2) You have to specify function signatures explicitly.
3) Each exported function can have only one signature (but you can export several different signatures under different names).
4) AOT compilation produces generic code for your CPU’s architectural family (for example “x86-64”), while JIT compilation produces code optimized for your particular CPU model.

Warning :
your codeas-is seems to be prone to fall into Amdahl's Law Trap

you will easily happen to pay way more on add-on overheads, than will ever receive back (from a (just) potential processing speedup - as more intensive memory-I/O-s and CPU-core cache re-use mechanics related constraints (both the absolute I/O ceilings and the relative losses of re-use efficiencies) do and will still work against your wish to get performance boosted )

At least, you have been warned :o)

user_36661970
Related