Equal Pandas DataFrames generate different pickle files

Viewed 72

Can anyone help me understand why I am getting different pickle files when I save (apparently) equal DataFrames?

My study case is:

  1. Create DataFrame & save to pickle
  2. Adding a new level to the columns, turning it into a multiIndex
  3. Dropping the newly added level to return to the original DataFrame & save to pickle
  4. The 2 DataFrames seem to be equal however, the 2 pickle files are different.

Here are the code snippets:

  1. Creating the test DataFrame:
import pandas as pd

df1 = pd.DataFrame(
    np.arange(6).reshape(2,3),
    index=['a', 'b'],
    columns=['A', 'B', 'C'],
)
df1.to_pickle('df1.pkl')

print(df1)
    A   B   C
a   0   1   2
b   3   4   5
  1. Adding a new level to the columns:
df2 = pd.concat({'new': df1}, names=['level'], axis=1)

print(df2)
level   new
        A   B   C
   a    0   1   2
   b    3   4   5
  1. Dropping the newly added level:
df3 = df2.droplevel('level', axis=1)
df3.to_pickle('df3.pkl')

print(df3)
    A   B   C
a   0   1   2
b   3   4   5

4.a. Testing the DataFrames:

df1.equals(df3)
True

4.b. From within a (OS) terminal, the pickle files are different:

$ diff df1.pkl df3.pkl
Binary files df1.pkl and df3.pkl differ

They actually have different sizes!

> dir
24/09/2021  13:59               763 df1.pkl
24/09/2021  13:59               761 df3.pkl

Any light on the apparent discrepancy would be greatly appreciated. Thanks

0 Answers
Related