How to deserialize a PyTorch saved model with private methods inside a class?

user_3108967 0
Viewed 996

I used the PyTorch saving method to serialize a bunch of essential objects. Among those, there was one class referencing a private method inside the __init__ of that same class. Now, after the serialization, I can't deserialize (unpickle) files because the private method is not accessible outside the class. Any idea how to solve or bypass it? I need to recover the data saved into the attributes of that class.

  File ".conda/envs/py37/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3331, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-1-a5666d77c70f>", line 1, in <module>
    torch.load("snapshots/model.pth", map_location='cpu')
  File ".conda/envs/py37/lib/python3.7/site-packages/torch/serialization.py", line 529, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File ".conda/envs/py37/lib/python3.7/site-packages/torch/serialization.py", line 702, in _legacy_load
    result = unpickler.load()
AttributeError: 'Trainer' object has no attribute '__iterator'
  • EDIT-1:

Here there is a piece of code that will generate the problem I’m facing right now.

import torch

class Test:
    def __init__(self):
        self.a = min
        self.b = max
        self.c = self.__private  # buggy

    def __private(self):
        return None

test = Test()

torch.save({"test": test}, "file.pkl")
torch.load("file.pkl")

However, if you remove the private attribute from the method, you won’t get any error.

import torch

class Test:
    def __init__(self):
        self.a = min
        self.b = max
        self.c = self.private  # not buggy

    def private(self):
        return None

test = Test()

torch.save({"test": test}, "file.pkl")
torch.load("file.pkl")
pytorch pickle deployment serialization python
2 Answers
user_63526770

This problem is due to name mangling — where the interpreter changes the name of the variable in below way which makes it harder to create collisions when the class is extended later. where

self.__private

has been changed to (self._className__privateMethodName)

self._Test__private

As name mangling is not applied with dunder, where a name has to start and end with double underscores.

So, to avoid name mangling add two more underscores at the end.

Below snippet should solve your problem.

import torch

class Test:
    def __init__(self):
        self.a = min
        self.b = max
        self.c = self.__private__

    def __private__(self):
        return None

test = Test()

torch.save({"test": test}, "file.pkl")
torch.load("file.pkl")
user_123824440
Related