Cannot pickle object: maximum recursion depth exceeded

user_14271703 0
Viewed 2549

I'm trying to pickle objects that I generate in a script to process them afterwards, but I get this error:

File "<ipython-input-2-0f716e86ecd3>", line 1, in <module>
pickle.dump(thisperson, output, pickle.HIGHEST_PROTOCOL)
RecursionError: maximum recursion depth exceeded while pickling an object

Here thisperson is relatively simple (or so I thought): a name and a dictionary containing attributes:

class Attributes:
    def __init__(self, what_attr, attr):
        self._what_attr = what_attr
        self._attributes = attr

    @property
    def what_attr(self):
        return self._what_attr
    @what_attr.setter
    def what_attr(self, new_value):
        self._what_attr = new_value

    @property
    def attributes(self):
        return self._attributes
    @attributes.setter
    def attributes(self, new_value):
        self._attributes = new_value


class Person:
    def __init__(self, watname, watage):
        self._myname = watname
        self._age = watage
        self._attributes = []

    @property
    def attributes(self):
        return self._attributes
    @attributes.setter
    def attributes(self, attributes_list):
        self._attributes = attributes_list

    @property
    def myname(self):
        return self._myname
    @myname.setter
    def myname(self, value):
        self._myname = value

    @property
    def age(self):
        return self._age
    @age.setter
    def age(self, value):
        self._age = value

the pickle function looks like this:

import pickle

def save_person(person, mypath):
    import os
    if not os.path.exists(mypath):
        os.makedirs(mypath)
    my_path = mypath + str(person.myname)
    if not os.path.exists(my_path + '/'):
        os.makedirs(my_path + '/')
    with open(my_path + '.pkl', 'wb') as output:
        pickle.dump(person, output, pickle.HIGHEST_PROTOCOL)

This error seems to be coming from the fact that the class contains a string and a dict because if I keep attributes as an empty dict I don't get the error.

Is there another way I can pickle objects like these?

I tried increasing sys.setrecursionlimit, but that didn't help.

Edit: .... i can't reproduce the error if i use a short version of the code .... go figure. @MinhNguyen something weird is that i create my attributes dict in a function (attr = get_attributes(data)), but when i create the same dict myself, i can pickle it ....

thisperson = Person("Kenny", 22, {})
attr = get_attributes(data)  # returns {'eyes': 'blue', 'hair': 'brown'}
this_attr = Attributes(person_name, attr)
this_person.attributes.append(this_attr)
save_person(this_person, main_folder)

Exception ignored in: 
'_pydevd_frame_eval.pydevd_frame_evaluator_win32_37_64.get_bytecode_while_frame_eval'
 RecursionError: maximum recursion depth exceeded while calling a Python object
 Process finished with exit code -1073741819 (0xC0000005)

but if i do the following it works:

 attr = {'eyes': 'blue', 'hair': 'brown'}
 this_attr = Attributes(person_name, attr)
 this_person.attributes.append(this_attr)
 save_person(this_person, main_folder)

Solution

Alright so after further digging, i found here that pickle doesnt mingle with dict well and will often crash.

So the workaround was that i dropped the Attribute class completely and now my Person's _attribute is a string of dict self._attributes = str(dict()) and whenever i want to add things to it, i'll do an eval(), update() and str() again.

This seem to be working so far...

pickle python
2 Answers

Problem

I've been using the concurrent.futures module, particularly the ProcessPoolExecutor class, which uses (kind of under the hood) pickle module to return values to the main process. The issue with this, is that pickle module seems to have a limit on what you can "pickle-ize" when we are talking about nested stuff. In my case, I had a structure like the following:

data = [
    {
        "string_list": [
            ["a1", "a2", ..., "an"],
            ["b1", "b2", ..., "bn"],
            ...
            ["n1", "n2", ..., "nn"],
        ]
    },
    {
        "string_list": [
            ["a1", "a2", ..., "an"],
            ["b1", "b2", ..., "bn"],
            ...
            ["n1", "n2", ..., "nn"],
        ]
    },

Basically, it was a list of dictionaries, which everyone of them contained, a list of string lists. I mean, it was a relatively nested structure. And every time i tried to get the data from the ProcessPoolExecutor's result, it raised a RecursionError: maximum recursion depth exceeded while pickling an object, due to pickle module, which can't (apparently) deal with this level of nesting.

Solution

My solution was similar to the provided in the edited question. I "stringified" the structure before pickling it, and then undone it once I could access the data. Personally, I used json module, which also comes with Python Standard Library.

This was my "experience" dealing with the pickle module.

user_161684210

Quick remark: I found that for my object, IntEnum caused this. Had one instance of it, replaced it with a bunch of constants, and the exception disappeared.

user_50124210
Related