I would like to iterate over a list of numpy arrays with different dimensions, and pass them in to a cython function that does not require the GIL:
# a has T1 rows and M columns
a = np.array([[0.0, 0.1, 0.3, 0.7],
[0.1, 0.2, 0.1, 0.6],
[0.1, 0.2, 0.1, 0.6]])
# b has T2 rows and M columns
b = np.array([[1.0, 0.0, 0.0, 0.0],
[0.1, 0.2, 0.1, 0.6]])
# c has T3 rows and M columns
c = np.array([[0.1, 0.0, 0.3, 0.6],
[0.5, 0.2, 0.3, 0.0],
[0.0, 1.0, 0.0, 0.0],
[0.0, 0.0, 0.1, 0.0]])
array_list = [a, b, c]
N = len(array_list)
# this function works
@cython.boundscheck(False)
@cython.wraparound(False)
cdef void function_not_requiring_the_gil(double[:, ::1] arr) nogil:
cdef int T = arr.shape[0]
cdef int M = arr.shape[1]
cdef int i, t
for t in range(T):
for i in range(M):
# do some arbitrary thing to the array in-place
arr[t, i] += 0.001
# issue is with this function
def function_with_loop_over_arrays(array_list, int N):
cdef int i
with nogil:
for i in range(N):
function_not_requiring_the_gil(array_list[i])
When I compile the Cython code, I get the following error:
Error compiling Cython file:
------------------------------------------------------------
...
def function_with_loop_over_arrays(array_list, int N):
cdef int i
with nogil:
for i in range(N):
function_not_requiring_the_gil(array_list[i]) ^
------------------------------------------------------------
my_file.pyx:312:53: Indexing Python object not allowed without gil
Is there another type of data structure that I can use instead of a Python list to store these numpy arrays so that I can iterate over them without the gil? I'm open to suggestions involving malloc C pointers/Cython memoryviews/other types I'm not aware of.
Note that each numpy array has a different number of rows, but the length of the list of arrays is known.