Question about deserializing some numbers (bug??)

user_14526807 0
Viewed 103

In order to deserialize bytes object, we use pickle.loads()

import pickle
import numpy as np
pickle.loads(np.float64(0.34103))

and the expected result is like below (because np.float64(0.34103) is not bytes objects, appropriate errors are expected)

---------------------------------------------------------------------------
UnpicklingError Traceback (most recent call last)
<ipython-input-19-5c07606a60f1> in <module>
----> 1 pickle.loads(np.float64(0.34103))

UnpicklingError: invalid load key, '\xc1'.

Here we have some questions that some numbers (it is rare) like 0.34104 prints the following result without errors.

pickle.loads(np.float64(0.34104))

True

This occurs only when the converted bytes start with b'\x88 (for example 0.04263, 0.08526, 0.11651 ...)

np.float64(0.34104).tobytes()

b'\x88.\xa8o\x99\xd3\xd5?'

Can anyone answer whether this issue is Python bugs?

Any answer will be highly appreciated.

pickle numpy python
2 Answers

pickle.loads's input isn't quite restricted to be a bytes object. Quoting the docs,

Return the reconstituted object hierarchy of the pickled representation data of an object. data must be a bytes-like object.

and quoting the "bytes-like object" docs,

An object that supports the Buffer Protocol and can export a C-contiguous buffer.

The buffer protocol is a way for objects to expose their underlying memory buffer to other code, for objects where that makes sense. NumPy arrays support the buffer protocol, allowing other code to operate on the underlying storage of an array instead of going through expensive Python-level indexing operations and wrapper objects. NumPy scalars support the buffer protocol too, to make working with 0-dimensional data as seamless as possible.

The error with pickle.loads(np.float64(0.34103)) isn't because np.float64(0.34103) isn't bytes. If you tried to pass in an ordinary float, you'd get a TypeError because the type is invalid (floats aren't bytes-like), but with np.float64(0.34103), the error you get is because pickle.loads tries to read the buffer of np.float64(0.34103) and finds that the data in there isn't a valid pickle.

With np.float64(0.34104), the buffer's contents do happen to be a valid pickle. By fluke, the bytes just happen to match a NEWTRUE opcode, a STOP opcode, and trailing garbage that gets ignored. NEWTRUE pushes a True on the pickle stack, STOP stops pickle execution, and the True gets returned.

user_23571120

This is not a Python bug. pickle.loads works with any bytes-like object, and a numpy float is one.
From the on this is just how bytes work: bytes represent raw data without type. That the 8 bytes representing numpy float 0.34104 happen to match some 8 bytes representing Python pickle True is a mere coincidence.

For pickle protocol 2 in specific, b'\x88.\xa8o\x99\xd3\xd5?' happens to be the opcode for True, End-Of-Pickle, and trailing garbage that is ignored.

>>> pickletools.dis(b'\x88.\xa8o\x99\xd3\xd5?')
    0: \x88 NEWTRUE
    1: .    STOP

The byte pattern of "numpy float 0.34104" for example also matches the little endian unsigned/signed Python integer 4599815250385579656, the big endian unsigned Python integer 9812965835362522431, the big endian signed Python integer -8633778238347029185, the latin1 string '\x88.¨o\x99ÓÕ?' and many other values.

>>> int.from_bytes(np.float64(0.34103), byteorder='big', signed=False)
13915064561919317311
user_53499160
Related