Why does pickling and loading a non-empty list change its size?

user_9736176 0
Viewed 208

I was looking at the sizes of pickled objects and noticed that non-empty lists change size after unpickling. They grow larger by 24 bytes. The size of empty lists stays the same. If I use the getsizeof method here, then it shows that the same happens with nested lists, and the size grows by 24 bytes for each non-empty list.

How does this increase happen?

A small example:

import pickle
import sys

li = [1]    

with open('test.p', 'wb') as f:
    pickle.dump(li, f)
print (getsize(li), sys.getsizeof(li))

with open('test.p', 'rb') as f:
    li2 = pickle.load(f)
print (sys.getsizeof(li2))
pickle python
1 Answers

The rebuilding process for pickled lists is different from the "building from scratch" process for list literals. When you have a literal list, it sizes it for that initial size precisely. When it's rebuilding from a pickle, it creates an empty list, then appends items one by one as they're unpickled, with overallocation occurring every time capacity is exhausted.

You see the size discrepancy from comparing to a manually append built list (because for all practical purposes, that's what unpickling is doing too):

import pickle
import sys

literal = [1]
incremental = []
incremental.append(1)
pickled = pickle.loads(pickle.dumps(literal, -1))

print("Literal:", sys.getsizeof(literal))
print("Incremental:", sys.getsizeof(incremental))
print("Pickled:", sys.getsizeof(pickled))

Try it online!

which on TIO produces:

Literal: 80
Incremental: 104
Pickled: 104

The numbers vary by interpreter (my own Python build gets 64, 88, 88), but the pattern is the same; over-allocation (to achieve O(1) amortized append costs) affects incremental/pickle based list construction, but not list literals.

user_3646960
Related