Subclass object of pathlib.Path gets custom attributes lost after pickle.load

user_10336496 0
Viewed 224
from pathlib import Path
import pickle
class P(type(Path())):
    def __init__(self, *args):
        super().__init__()
        self.a = ''
p = P()
p.a = 'x'
with open('xx', 'wb') as wf:
    pickle.dump(p, wf)
p1 = pickle.load(open('xx', 'rb'))
print(p1.a)            # here p1.a is ''

I am making a subclass of pathlib.Path and like to add some custom attributes to it. The problem is that the custom attribute gets lost after reloaded by pickle. How to solve this problem.

Other solutions I tried:

  • use __slots__, the same issue.
  • use composition instead of inheritance, then dispatch Path-like methods by implementing __getattr__. However, in this case, self.path is not initialized within pickle.load thus causing an endless call of __getattr__.
class File():
    def __init__(self, *args):
        self.path = Path(*args)
    def __getattr__(self, item):
        return getattr(self.path, item)
p = File('aaa')
p.exists()  # no error
with open('xx', 'wb') as wf:
    pickle.dump(p, wf)
p1 = pickle.load(open('xx', 'rb')) 
# RecursionError: maximum recursion depth exceeded. 
# This is due to call of self.path, in that moment, path is not in self.__dict__
pickle python-3.x serialization inheritance python
2 Answers

One way to do it would be by using the copyreg module to associate a pickle support function to instance of your class as shown below. Note I also had to modify how your P class handles arguments — it no longer ignores them.

import copyreg
from pathlib import Path
import pickle


class P(type(Path())):
    def __init__(self, *args):
        super().__init__()
        self.a = args[0] if args else ''


def pickle_P(p):
    print("pickling a P instance...")
    return P, (p.a,)

copyreg.pickle(P, pickle_P)

p = P()
p.a = 'x'
q = P('y')

with open('xx', 'wb') as outp:
    pickle.dump(p, outp)
    pickle.dump(q, outp)

with open('xx', 'rb') as inp:
    p1 = pickle.load(inp)
    q1 = pickle.load(inp)

print('p1.a = {!r}'.format(p1.a))
print('q1.a = {!r}'.format(q1.a))

Output:

pickling a P instance...
pickling a P instance...
p1.a = 'x'
q1.a = 'y'
user_3552300

For the problem of the inheritance one.

Another solution as a note to @martineau’s answer.

If I'm correct, the problem is caused by the __reduce__ method in pathlib.PosixPath. It seems like that the pickle behavior will be determined by this method. and @martineau's solution which uses copyreg.pickle(P, pickle_P) is also related to this method: pickle_P has the same return pattern as __reduce__.

Here is the doc about return values of __reduce__:

When a tuple is returned, it must be between two and six items long. Optional items can either be omitted, or None can be provided as their value. The semantics of each item are in order:

  • A callable object that will be called to create the initial version of the object.

  • A tuple of arguments for the callable object. An empty tuple must be given if the callable does not accept any argument.

  • Optionally, the object’s state, which will be passed to the object’s __setstate__() method as previously described. If the object has no such method then, the value must be a dictionary and it will be added to the object’s __dict__ attribute.

  • ...

The second item explains how does @martineau's solution works: the second return value will be passed into __init__.

And this is the source code of PosixPath.__reduce__

    def __reduce__(self):
        # Using the parts tuple helps share interned path parts
        # when pickling related paths.
        # self._parts is arguments passed to Path
        return (self.__class__, tuple(self._parts))

According to the description of the third return value, the solution is:

from pathlib import Path
import pickle
class P(type(Path())):
    def __init__(self, *args):
        super().__init__()
        self.a = ''
    def __reduce__(self):
        return self.__class__, tuple(self._parts), self.__dict__

p = P()
p.a = 'x'
with open('xx', 'wb') as wf:
    pickle.dump(p, wf)
p1 = pickle.load(open('xx', 'rb'))
print(p1.a)            # here p1.a is 'x'

Shortcomings of this solution:

  • P's instance will contain a __dict__ attribute (Path uses __slots__).
  • attribute named _hash will be ignored.

For the problem of the composition one

A note in the doc of pickle may explain the cause of the error.

Note At unpickling time, some methods like __getattr__(), __getattribute__(), or __setattr__() may be called upon the instance. In case those methods rely on some internal invariant being true, the type should implement __new__() to establish such an invariant, as __init__() is not called when unpickling an instance.

To ensure path attribute exits when calling __getattr__, a solution is to move the attribute assignment into __new__ method (before __init__).

class File():
    def __new__(cls, *args):
        obj = super().__new__(cls)
        obj.path = Path(*args)
        return obj
    def __getattr__(self, item):
        return getattr(self.path, item)
p = File('aaa')
p.exists()  # no error
with open('xx', 'wb') as wf:
    pickle.dump(p, wf)
p1 = pickle.load(open('xx', 'rb')) # no error
user_103364960
Related