Can anyone help me understand why I am getting different pickle files when I save (apparently) equal DataFrames?
My study case is:
- Create DataFrame & save to pickle
- Adding a new level to the columns, turning it into a multiIndex
- Dropping the newly added level to return to the original DataFrame & save to pickle
- The 2 DataFrames seem to be equal however, the 2 pickle files are different.
Here are the code snippets:
- Creating the test DataFrame:
import pandas as pd
df1 = pd.DataFrame(
np.arange(6).reshape(2,3),
index=['a', 'b'],
columns=['A', 'B', 'C'],
)
df1.to_pickle('df1.pkl')
print(df1)
A B C
a 0 1 2
b 3 4 5
- Adding a new level to the columns:
df2 = pd.concat({'new': df1}, names=['level'], axis=1)
print(df2)
level new
A B C
a 0 1 2
b 3 4 5
- Dropping the newly added level:
df3 = df2.droplevel('level', axis=1)
df3.to_pickle('df3.pkl')
print(df3)
A B C
a 0 1 2
b 3 4 5
4.a. Testing the DataFrames:
df1.equals(df3)
True
4.b. From within a (OS) terminal, the pickle files are different:
$ diff df1.pkl df3.pkl
Binary files df1.pkl and df3.pkl differ
They actually have different sizes!
> dir
24/09/2021 13:59 763 df1.pkl
24/09/2021 13:59 761 df3.pkl
Any light on the apparent discrepancy would be greatly appreciated. Thanks