Turn off list reflection in Numba

user_2475758 0
Viewed 8152

I'm trying to accelerate my code using Numba. One of the arguments I'm passing into the function is a mutable list of lists. When I try changing one of the sublists, I get this error:

Failed in nopython mode pipeline (step: nopython mode backend) cannot reflect element of reflected container: reflected list(reflected list(int64))

I don't actually care about reflecting changes I make to the native list into the original Python list. How do I go about telling Numba not to reflect the changes? The documentation is pretty vague regarding list reflection in Numba.

Thanks,

numba python
2 Answers

Quoting directly from the docs:

In nopython mode, Numba does not operate on Python objects. list are compiled into an internal representation. Any list arguments must be converted into this representation on the way in to nopython mode and their contained elements must be restored in the original Python objects via a process called reflection.

Reflection is required to maintain the same semantics as found in regular Python code. However, the reflection process can be expensive for large lists and it is not supported for lists that contain reflected data types. Users cannot use list-of-list as an argument because of this limitation.

Your best bet would be to give a 2D numpy array of shape len(ll) x max(len(x) for x in ll), ll being the list of lists. I myself use something like this to achieve this, and then pass the arr, lengths to the njit compiled function:

def make_2D_array(lis):
    """Funciton to get 2D array from a list of lists
    """
    n = len(lis)
    lengths = np.array([len(x) for x in lis])
    max_len = np.max(lengths)
    arr = np.zeros((n, max_len))

    for i in range(n):
        arr[i, :lengths[i]] = lis[i]
    return arr, lengths

HTH.

user_39245660

If you pass list of list parameter to numba, you should use numpy array instead of original Python list. Numba raise reflection error because of the not supported list features. You can compare the two examples below:

This one is getting the same error:

TypeError: Failed in nopython mode pipeline (step: nopython mode backend)
cannot reflect element of reflected container: reflected list(reflected list(int64))

import numba

list_of_list = [[1, 2], [34, 100]]


@numba.njit()
def test(list_of_list):
    if 1 in list_of_list[0]:
        return 'haha'

test(list_of_list)

Smooth running version is;

from numba import njit
import numpy as np


@njit
def test():
    if 1 in set(np_list_of_list[0]):
        return 'haha'


if __name__ == '__main__':
    list_of_list = [[1, 2], [34, 100]]
    np_list_of_list = np.array(list_of_list)
    print(test())
user_109370310
Related