how to use Apache Arrow to do "a + b + c*5 + d*3"?

user_10036027 0
Viewed 121

I got the idea of using pre-defined functions to do this: calculate "a + b", "c * 5", "d * 3" and then add the result.

But this way seems generate a lot of code. Is there any better methods to do this?

By the way, does Apache Arrow use SIMD by default(c++ version)? If not, how can I make it use SIMD?

apache-arrow c++
1 Answers

PyArrow doesn't currently override operators in Python, but you can easily call the arithmetic compute functions. (functools.reduce is used here since the addition kernel is binary, not n-ary.)

PyArrow automatically uses SIMD, based on what flags it was compiled with. It should use the 'highest' SIMD level supported by your CPU for which it was compiled with. Not all compute function implementations leverage SIMD internally. Right now it looks like it's mostly the aggregation kernels which do so.

>>> import pyarrow as pa
>>> import pyarrow.compute as pc
>>> import functools
>>> pa.__version__
'4.0.1'
>>> a = pa.array([1,2,3])
>>> b = pa.array([3,4,5])
>>> c = pa.array([1,0,1])
>>> d = pa.array([2,4,2])
>>> functools.reduce(pc.add, [pc.add(a,b), pc.multiply(c, 5), pc.multiply(d, 3)])
<pyarrow.lib.Int64Array object at 0x7fd5a0d9c040>
[
  15,
  18,
  19
]
user_2627270
Related