I compared how long it takes to add two vectors in pandas/numpy/pure python and got some surprising (for me) results.
Tested on python3.6.9, Ubuntu 18.04, numpy=1.18.1, pandas==1.1.3.
CODE
import pandas as pd
import numpy as np
import random
from timeit import timeit
repeat = 1000
max_exp =7
def time_all(a, b):
sa = pd.Series(a)
sb = pd.Series(b)
na = np.array(a)
nb = np.array(b)
py_sum = lambda: [x + y for x, y in zip(a, b)]
pd_sum = lambda: sa + sb
np_sum = lambda: na + nb
py_time = timeit(py_sum, number=repeat)
pd_time = timeit(pd_sum, number=repeat)
np_time = timeit(np_sum, number=repeat)
return py_time, pd_time, np_time
for i in range(2, max_exp):
size = 10 ** i
a = [random.randint(0, 100) for x in range(0, size)]
b = [random.randint(0, 100) for x in range(0, size)]
py_time, pd_time, np_time = time_all(a, b)
py_pd = round(py_time / pd_time, 4)
py_np = round(py_time / np_time, 4)
pd_np = round(pd_time / np_time, 4)
print('''
ARRAY SIZE: {}
PY/PD: {}
PY/NP: {}
PD/NP: {}'''.format(size, py_pd, py_np, pd_np))
RESULT
ARRAY SIZE: 100
PY/PD: 0.0357
PY/NP: 8.656
PD/NP: 242.1978
ARRAY SIZE: 1000
PY/PD: 0.3286
PY/NP: 44.3758
PD/NP: 135.05
ARRAY SIZE: 10000
PY/PD: 2.8774
PY/NP: 69.2828
PD/NP: 24.0785
ARRAY SIZE: 100000
PY/PD: 11.7784
PY/NP: 76.5296
PD/NP: 6.4974
ARRAY SIZE: 1000000
PY/PD: 26.2985
PY/NP: 33.1973
PD/NP: 1.2623
QUESTIONS
- For small arrays (up to ~5k elements) python is more efficient than pandas, for very small the difference is big. Is there any general rule what operations have (so much) better performance on pure-python than pandas? Something like "for less then 100 rows pandas will be order of magnitude slower whatever you do"?
- As far as I understand, pandas internally uses numpy. Do those performances differ only because of conversion time between pd.Series and np.array? If yes, is there any way to force the conversion when Series is created?
- (least important, but most surprising) For 10^6 array pure python was ~ 77 times slower than numpy, but for 10^7 the ratio is only 33. Why?