protocol=4 pickle (python 3.7): Keyerror when load dicts with the same key inside

user_15010818 0
Viewed 612

I'm running python3.7.9-64bit installed by anaconda on Mac.
Just happens to meet three problems.

(Though I write fix_import=False, changing this does not make any difference)
1st Problem, if two dicts with the same key got pickled in one file, loading the 2nd dict will raise KeyError.

import pickle

d1={"a":1}# if two dicts has the same key
d2={"a":0}# does not matter what value is stored.

tmpf=open("deleteThis","wb")
pkl=pickle.Pickler(tmpf,protocol=4,fix_imports=False)
pkl.dump(d1)
pkl.dump(d2)
tmpf.close()

tmpf2=open("deleteThis","rb")
td1=pickle.load(tmpf2)
td2=pickle.load(tmpf2)# KeyError 1

2nd Problem, though it probably relates to the first one, if the 2nd dict has a dict D inside it which share the same key with the 1st dict, then D's key changes when loaded.

d1={"a":1}
d2={"lll":{"a":2} }# this "a" will always be "lll"

tmpf=open("deleteThis","wb")
pkl=pickle.Pickler(tmpf,protocol=4,fix_imports=True)
pkl.dump(d1)
pkl.dump(d2)
tmpf.close()

tmpf2=open("deleteThis","rb")
td1=pickle.load(tmpf2)
td2=pickle.load(tmpf2)
print(td1,td2)# {'a': 1} {'lll': {'lll': 2}}

3rd Problem, and the most devastating to me, if there is a dict get pickle.dump and the 2nd dumped dict has a list of dicts sharing the same key, loading the 2nd dict raise KeyError.

d1={"b":0}
d2={ "l":[{"a":2} , {"a",3}] }

tmpf=open("deleteThis","wb")
pkl=pickle.Pickler(tmpf,protocol=4,fix_imports=True)
pkl.dump(d1)
pkl.dump(d2)
tmpf.close()

tmpf2=open("deleteThis","rb")
td1=pickle.load(tmpf2)
td2=pickle.load(tmpf2)# KeyError

These problems only happens with pickle( protocol=4 ).
Nothing goes wrong with protocol=3
Unknown about python>3.8 or protocl=5

pickle python-3.x
1 Answers

The short answer to your question is to use pickle.Unpickler() to load your pickles.

Alternatively, don't use pickle.Pickler(). Instead write each pickle with pickle.dump() and read back with pickle.load() or pickle.Unpickler().

Either one of those should cure your problem.

I can confirm that the same problem that you describe exists in Python 3.9.1 for both protocol versions 4 and 5.

BTW: notice that {"a",3} in your last example is a set, not a dict as you thought. Nevertheless the same error will occur.

The problem is that Pickler uses a memo to cache data that it has pickled. It uses this to economise on the size of the resulting file by avoiding storing the same data more than once. The memo is shared between all pickles written with the Pickler.

The Unpickler uses the memo to reconstruct the pickled objects that share cached data. However, pickle.load() does not use the memo and can therefore fail to find the values that the Pickler memoized when it dumped the individual pickles.

Here is some code to demonstrate:

import pickle

d1 = {"b": 0}
d2 = {"l": [{"a": 2}, {"a": 3}]}

with open("deleteThis", "wb") as tmpf:
    pkl = pickle.Pickler(tmpf, protocol=4)
    for obj in d1, d2:
        pkl.dump(obj)

# reload objects with Unpickler - will work.
with open("deleteThis", "rb") as tmpf:
    unpkl = pickle.Unpickler(tmpf)
    print('Using Unpickler')
    while True:
        try:
            print(unpkl.load())
        except EOFError:
            break

# reload objects with pickle.load - might work, but won't here.
with open("deleteThis", "rb") as tmpf:
    print('Using pickle.load()')
    while True:
        try:
            print(pickle.load(tmpf))
        except EOFError:
            break

And it's output:

Using Unpickler
{'b': 0}
{'l': [{'a': 2}, {'a': 3}]}
Using pickle.load()
{'b': 0}
Traceback (most recent call last):
  File "/tmp/z.py", line 26, in 
    print(pickle.load(tmpf))
_pickle.UnpicklingError: Memo value not found at index 6
mhawke_219450
Related