Numba with list of tuples

Question

I have a list of tuples queried from a db.

tuple_data = [(1.1,1,"one"),(2.1,2,"two"),(3.1,3,"three")]

each tuple will contain different data-types.

from this list i need 1st element from each tuple, so i did:

data = [result[0] for result in tuple_data]

Now i am trying to use numba module instead of list comprehension.

So i tried below method:

@numba.njit(cache = True)
def loop_faster(results):
    res = []
    for result in results:
        res.append(result[0])

This throws NumbaPendingDeprecationWarning: , i am not able to use list of tuples in iteration (as per numba docs)

So i changed it to numpy array (From here):

L_arr = np.array(tuple_data)

now everything is fine.loop_fastermethod works.

The catch is , my original data is (float, int, str) while changing to numpy array its all (str,str,str) which is expected.

The problem is i want the data as float itself.

So my code goes like:

import numba, logging
import numpy as np

numba_logger = logging.getLogger('numba')
numba_logger.setLevel(logging.WARNING)

@numba.njit(cache = True)
def loop_faster_1(results, n):
    res = []
    for result in results:
        res.append(result[0])
    print(res)

t1 = [(1.1,1,"one"),(2.1,2,"two"),(3.1,3,"three")]
L_arr = np.array(t1)
loop_faster_1(L_arr,0)

In real scenario my tuple list is huge, i convert it to numpy array for numba and again i need the data in float so i have to convert all str to float.

Basically with numba,

List of tuples
convert to numpy array
call numba method
convert back to float
use for further processing.

but with list comprehension,

List of tuples
List comprehension
use for further processing.

Is there a better way to do this using numba? or i just go with list comprehension to remove these steps while using numba. Because with this i feel i am actually killing the very purpose of reducing time taken for loops.

user_901925 · Answer

You shouldn't need numba to get the first column of a numpy array fast. That's a basic indexing operation.

Your sample:

In [23]: tuple_data = [(1.1,1,"one"),(2.1,2,"two"),(3.1,3,"three")]

The obvious list comprehension:

In [24]: [x[0] for x in tuple_data]
Out[24]: [1.1, 2.1, 3.1]

The array approach:

In [25]: np.array(tuple_data)[:,0].astype(float)
Out[25]: array([1.1, 2.1, 3.1])

The list comprehension is much faster - for this small sample:

In [26]: timeit [x[0] for x in tuple_data]
349 ns ± 7.76 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
In [27]: timeit np.array(tuple_data)[:,0].astype(float)
15.4 µs ± 55.4 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

The big time consumer is np.array(...). The indexing is fast. The comprehension should scale linearly with list size. But so does np.array, after all it is processing the list element by element. There might a size where it ends up faster, but past experience suggests that's with 1000+ tuples.

An alternative is to use a compound dtype, making a structured array. Now the floats aren't converted to strings. But it doesn't improve timing.

In [28]: np.array(tuple_data, dtype='f,f,U10')
Out[28]: 
array([(1.1, 1., 'one'), (2.1, 2., 'two'), (3.1, 3., 'three')],
      dtype=[('f0', '


Here's another approach - make an object dtype array:
In [31]: np.array(tuple_data, dtype=object)[:,0]
Out[31]: array([1.1, 2.1, 3.1], dtype=object)
In [32]: timeit np.array(tuple_data, dtype=object)[:,0]
4.52 µs ± 9.41 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

That's better than the other arrays, those still behind the comprehension.  It makes a (3,3) array of objects, the same ones in the tuples.  It's very list-like, except it does multidimensional indexing.
The full arrays:
In [33]: np.array(tuple_data, dtype=object)
Out[33]: 
array([[1.1, 1, 'one'],
       [2.1, 2, 'two'],
       [3.1, 3, 'three']], dtype=object)
In [34]: np.array(tuple_data, dtype='f,f,U10')
Out[34]: 
array([(1.1, 1., 'one'), (2.1, 2., 'two'), (3.1, 3., 'three')],
      dtype=[('f0', '

Numba with list of tuples

1 Answers