How can I change the definition of class methods after pickling

user_13771479 0
Viewed 216

I am writing a class that does some data processing. I initiated the class and did some processing then pickled the results. the problem is that I added new methods to the class and loaded the pickled objects but I couldn't apply the new methods to the unpickled ones.

For example, This is my class

class Play(ABC):
  def __init__(self, data):
    self.data = data
    
  @abstractmethod
  def method(self):
    pass

class M(Play):

  def __init__(self, data):
    super().__init__(data)

  def method(self):
    self.corr = np.corrcoef(self.data)

I did some processing

mat = np.random.rand(102, 641)
s = M(mat)
s.method()

and pickled them using dill

def save_object(obj, filename):
    with open(filename, 'wb') as output:  
        dill.dump(obj, output, dill.HIGHEST_PROTOCOL)
save_object(s, 'file.pkl')

then, I added new methods to the class and unpickled the file to apply those methods but I couldn't

class Play(ABC):
  def __init__(self, data):
    self.data = data
    
  @abstractmethod
  def method(self):
    pass

class M(Play):

  def __init__(self, data):
    super().__init__(data)

  def method(self):
    self.corr = np.corrcoef(self.data)
  
  def new(self):
    # pass
    self.q = self.corr.shape
def load_object(filename):
  with open(filename, 'rb') as input:
    obj = dill.load(input)
    return obj

obj = load_object('file.pkl')
obj.new()

and I get this result

AttributeError: 'M' object has no attribute 'new'

How can I fix that?

dill pickle python-3.x oop python
2 Answers

I'm the dill author. You may want to look at the different pickling settings (see dill.settings). For example:

>>> class Foo(object):
...   x = 1
...   def bar(self, y):
...     return y + self.x
... 
>>> import dill
>>> f = Foo()
>>> s = dill.dumps(f)
>>> f.bar(5)
6
>>> 
>>> class Foo(object):
...   x = 10
...   def bar(self, y):
...     return y + self.x**2
... 
>>> g = dill.loads(s)
>>> g.bar(5)
105
>>> g = dill.loads(s, ignore=True)
>>> g.bar(5)
6
>>> dill.settings
{'protocol': 4, 'byref': False, 'fmode': 0, 'recurse': False, 'ignore': False}
>>>

here, ignore=True on load tells dill to ignore an existing class definition if a newer one exists. The other settings are to be used on dump/dumps, where for example, byref=True tells dill to not store the class definition at all -- just use whatever references is available in the unpicking environment.

user_23794330

An alternative for you is to use joblib library. I will also encourage you to check the documentation for joblib shared in the References section below.

joblib.dump(obj, "filename.joblib")    ## Saving object to a file
obj = joblib.load("filename.joblib")   ## Loading object from a file

Save and Load your object: 1st time

class Foo(object):
    x = 1

    def bar(self, y):
        return y + self.x

f1 = Foo()

def checkFoo(foo: Foo):
    print(f'version: {foo.version}')

## save your object
>>> version = '0.0.1'
>>> f1.version = version
>>> checkFoo(f1)
'0.0.1'
>>> joblib.dump(f1, f'object_store_v{version}.joblib')
['object_store_v0.0.1.joblib']

## load saved object (latest version)
>>> version = '0.0.1'
>>> f2 = joblib.load(f'object_store_v{version}.joblib')
>>> checkFoo(f2)
'0.0.1'

Save and Load your object: 2nd time (after modification)

## Modify object and save new version
>>> version = '0.0.2'
>>> f2.version = version
>>> checkFoo(f2)
'0.0.2'
>>> joblib.dump(f2, f'object_store_v{version}.joblib')
['object_store_v0.0.2.joblib']

## load saved object (latest version)
>>> version = '0.0.2'
>>> f3 = joblib.load(f'object_store_v{version}.joblib')
>>> checkFoo(f3)
'0.0.2'

References

user_84748940
Related