Why are operations on pandas.DataFrames so slow?! Look at the following examples.
Measurement:
- Create a
numpy.ndarraypopulated with random floating point numbers - Create a
pandas.DataFramepopulated with the same numpy array
The I measure the time of the following operations
For the
numpy.ndarray- Take the sum along the 0-axis
- Take the sum along the 1-axis
For the
pandas.DataFrame- Take the sum along the 0-axis
- Take the sum along the 1-axis
For the
pandas.DataFrame.values -> np.ndarray- Take the sum along the 0-axis
- Take the sum along the 1-axis
Observations
- Summing over
numpy.ndarrays' is much faster then operating onpandas.DataFrames`. - This is even true, if the
pd.DataFramedoes not contain only floating point numbers and has nothing special attached (MultiIndex or whatever). - Operations on
numpy.ndarrayare about 7 to 10 times faster.
Questions
- Why does this happen?
- How can this be optimized?
- Is
pandasnot able to call or pass throughnumpys' operations?
import numpy as np
import pandas as pd
n = 50000
m = 5000
array = np.random.uniform(0, 1, (n, m))
dataframe = pd.DataFrame(array)
Numpy
%%timeit
array.sum(axis=0)
206 ms ± 3.78 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%%timeit
array.sum(axis=1)
233 ms ± 33.9 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
Pandas
%%timeit
dataframe.sum(axis=0)
1.65 s ± 14.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%%timeit
dataframe.sum(axis=1)
1.74 s ± 15.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Pandas without Pandas
Let's operate on the values alone ...
%%timeit
dataframe.values.sum(axis=0)
206 ms ± 7.13 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%%timeit
dataframe.values.sum(axis=1)
181 ms ± 1.66 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)