faster column-multiply in dataframe

Viewed 598

I have a pandas dataframe A that has 2 columns x and y. I want to multiply them like B = A['x'] * A['y']. Is there any faster way to do this? would A['a'].mul(A['y']) be faster?

3 Answers

To check which is faster you can check the time that it takes for each case: In Ipython or Jupiter would be:

%%timeit
    d['a'] * d['b']

For a dataframe like this one:

a = np.arange(0,10000)
b = np.ones(10000)

d = pd.DataFrame(np.vstack([a,b]).T, columns=["a","b"])

Get your multiplication:

1- in pandas

d['a'] * d['b']
81.2 µs ± 977 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

2 - in numpy. avoiding pandas overhead

d['a'].values * d['b'].values
9.21 µs ± 41.4 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

... If you are worried so much about speed, use just numpy. Take advantage of the nice feature of pandas to allow you to access the array with the feature values.

Numpy is much faster, you use np.arrays if your columns are lengthy

import numpy as np

B=np.array(A.x)*np.array(A.y)

A quick test on my PC on a dataframe with 55K rows reduced the time from 0.78 sec (your original method) to 0.54 sec (above method)

I could only agree with @IoaTzimas 's comment, Numpy is faster. So you'd be better off converting the Dataframe's columns that you want to multiply to Numpy arrays and work with them.

If you have the initial data in arrays, you can just convert them to a numpy array and use that.

If you have your data in a Dataframe and need to extract it beforehand you could do the following:

from numpy import multiply as npMultiply

# get the values from the Dataframe to arrays
x_array = A['x'].values
y_array = A['y'].values

B = npMultiply(x_array, y_array)
Related