I read an article (https://www.ritchieng.com/pandas-making-dataframe-smaller-faster/) which mentions that it makes the DataFrame faster by making it smaller (by converting data type).
Is there any association between smaller (memory usage) and faster (cpu time)?
Let's say I have a DataFrame column of int64. If we convert it to int8, will the operation on the column be faster? e.g. assuming that the operation is d[col] = d[col] + 1
Changing the data type from int64 to int8 reduces the amount of bits required to store the data. This will drastically reduce your memory usage which will be very helpful when you have large data for intense computation. This increases the size of data that's possible with pandas before you get a memory error.
What will also increase performance is changing object-type columns to specific types since this allows type-optimisations.
why not to test it?
In [29]: df = pd.DataFrame(np.random.randint(100, size=(10**7, 10), dtype="int64"))
In [30]: df.dtypes
Out[30]:
0 int64
1 int64
2 int64
3 int64
4 int64
5 int64
6 int64
7 int64
8 int64
9 int64
dtype: object
memory usage (in MiB):
In [31]: df.memory_usage().sum() / 1024**2
Out[31]: 762.9395294189453
timings:
In [32]: %timeit df.agg(["min","max","mean"])
4.68 s ± 84.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [33]: %timeit df+1
818 ms ± 26 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [34]: df2 = df.astype("int8")
In [35]: df2.dtypes
Out[35]:
0 int8
1 int8
2 int8
3 int8
4 int8
5 int8
6 int8
7 int8
8 int8
9 int8
dtype: object
memory usage (in MiB):
In [38]: df2.memory_usage().sum() / 1024**2
Out[38]: 95.36750793457031
timings:
In [36]: %timeit df2.agg(["min","max","mean"])
2.4 s ± 222 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [37]: %timeit df2+1
170 ms ± 10.6 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)