Pickle module that imports code with C bindings

user_1830334 0
Viewed 422

I have a class I want to pickle using this function:

def _pickle(self):
    """Pickle model instance.
    """
    fpath = f'{directory}/model.pickle'
    with open(fpath, 'wb') as f:
        pickle.dump(self, f, pickle.HIGHEST_PROTOCOL)

However, in my file I import a library, PyPolyaGamma. It's a Python wrapper for some C code that does fast sampling from the Polya-gamma distribution. When I try to pickle my class, I get this error:

Traceback (most recent call last):
  File "fit_model.py", line 262, in <module>
    model.fit(Y)
  File "/Users/gwg/projects/cdnlvm/kl_gplvmbase.py", line 97, in fit
    self._plot_and_print(t)
  File "/Users/gwg/projects/cdnlvm/kl_nbgplvm.py", line 456, in _plot_and_print
    self._pickle()
  File "/Users/gwg/projects/cdnlvm/kl_gplvmbase.py", line 352, in _pickle
    pickle.dump(self_, f, pickle.HIGHEST_PROTOCOL)
  File "stringsource", line 2, in pypolyagamma.pypolyagamma.PyPolyaGamma.__reduce_cython__
TypeError: no default __reduce__ due to non-trivial __cinit__

My understanding is that Python does not know how to serialize the PyPolyaGamma code, probably because of the dependency on C. What is the correct way to handle this?

Here is a minimal and complete example, if you're willing to install PyPolyaGamma:

# pickle_test.py

import pickle
from   pypolyagamma import PyPolyaGamma

class Model:
    def __init__(self):
        self.pg = PyPolyaGamma()

model = Model()
with open('test.pickle', 'wb+') as f:
    pickle.dump(model, f)

This will output:

Traceback (most recent call last):
  File "pickle_test.py", line 15, in <module>
    pickle.dump(model, f)
  File "stringsource", line 2, in pypolyagamma.pypolyagamma.PyPolyaGamma.__reduce_cython__
TypeError: no default __reduce__ due to non-trivial __cinit__
pickle cython python
2 Answers

Unfortunately it looks like that pypolyagamma library you use doesn't currently support pickling/unpickling those objects, so in a word: you can't pickle them without modifying the library.

Furthermore, since it wraps a C++ class, the pickler/unpickler isn't trivial; it'd need to know what state from that wrapped class is necessary for the object to be in the same state after unpickling.

AKX_516850

As an alternative to a wrapper and if you don't mind using a functional interface, then I would recommend using polyagamma instead. This resolves issues with pickling. Its a python package for Polya-Gamma variable sampling written in C. It is flexible and allows using different sampling methods and has an API that resembles Numpy's. You can install the latest version via pip install -U polyagamma.

user_119574010
Related