dill fails to pickle sympy.Float under Python 3.7 for dill.settings["recurse"]=True

user_19743379 0
Viewed 48

Under python 3.7, dill fails to pickle sympy.Float objects when the "recurse" setting is set to True:

dill.settings["recurse"] = True
import sympy
tst = sympy.Float(3.3)
dill.detect.errors(tst)

This fails with RecursionError('maximum recursion depth exceeded in comparison'). Does anybody know a workaround?

The error disappears for Python >=3.8, and also does not occur in the absence of the "recurse" setting. Unfortunately I need to support Python 3.7 and am forced to use the "recurse" mode.

dill pickle python
2 Answers

Change pickle protocol from 3 to 4:

import dill
dill.settings["recurse"] = True
dill.settings["protocol"] = 4  # Add this
import sympy
tst = sympy.Float(3.3)
dill.detect.errors(tst)

In Python 3.8 and above, pickle sets DEFAULT_PROTOCOL = 4.

Explanation

  1. In pickle save_reduce, if protocol < 4, a func is saved instead of the Float class:
if self.proto >= 4:
    save(cls)
    save(args)
    save(kwargs)
    write(NEWOBJ_EX)
else:
    func = partial(cls.__new__, cls, *args, **kwargs)
    save(func)
    save(())
    write(REDUCE)

  1. With "recurse" = True, dill recursively traces and pickles objects referred to in the global dictionary of functions (starting with cls.__new__, i.e. Float.__new__).

  2. sympy has circular references in Float.__new__ → _convert_numpy_types → sympify → _convert_numpy_types (repeated) → ...

Proof of concept

You can reproduce the error with:

def a():
    b

def b():
    a

import dill
dill.settings["recurse"] = True
dill.detect.errors(a)
user_86017600

I'm the dill author. I'm not seeing the reported behavior. Maybe you are using an older version of Python, dill, or sympy.

Python 3.7.13 (default, May 10 2022, 11:13:40) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> import sympy
>>> dill.__version__
'0.3.6.dev0'
>>> sympy.__version__
'1.10.1'
>>> tst = sympy.Float(3.3)
>>> dill.copy(tst)
3.30000000000000
>>> dill.settings['recurse'] = True
>>> dill.copy(tst)
3.30000000000000
>>> dill.detect.errors(tst)
>>>

Here, copy(tst) is the same as loads(dumps(tst)). If your versions are older than mine, and after you update you still see the same behavior (or it otherwise persists for you), then please do let me know in the comments (and fill out an issue on the dill GitHub page).

Note: We have a release that is imminent, so dill-0.3.6 will be out in a matter of days.

user_23794330
Related