Troubleshooting a custom "Typed Property" pattern that overwrites property data on subsequent class instantiations

user_2059999 0
Viewed 35

I'm trying to put together a utility that works kind of like Python's dataclass but gives me some additional things I can do like type checking on assignment, etc. I'm basically following the general pattern described in "9.21 Avoiding Repetitive Property Methods" from the O'Reilly Python Cookbook, 3rd Edition

I'm running into an issue where subsequent instantiations of MyClass overwrite data from other instances of it. Here's a simplified version of what I'm doing:

def typed_property(name, default=None):
    varname = "_" + name

    @property
    def prop(self):
        if not hasattr(self, varname):
            setattr(self, varname, default)
        return getattr(self, varname)

    @prop.setter
    def prop(self, value):
        setattr(self, varname, value)

    return prop

class MyClass(object):
    data = typed_property("data", default={})

If I run this using something like this:

obj1 = MyClass()
obj1.data["test"] = set()
obj1.data["test"].add(1)
print(">>> initial")
print("(1) id(obj1) =", id(obj1))
print("(2) obj1.data =", obj1.data)

print(">>> create obj2")
obj2 = MyClass()
obj2.data["test"] = set()

print("(3) obj1.data =", obj1.data) 
print("(4) obj2.data =", obj2.data)

print(">>> ID's:")
print("(5) id(obj1) =", id(obj1))
print("(6) id(obj2) =", id(obj2))
print("(7) id(obj1.data) =", id(obj1.data))
print("(8) id(obj2.data) =", id(obj2.data))

I get this output:

>>> initial
(1) id(obj1) = 4429860720
(2) obj1.data = {'test': {1}}
>>> create obj2
(3) obj1.data = {'test': set()}
(4) obj2.data = {'test': set()}
>>> ID's:
(5) id(obj1) = 4429860720
(6) id(obj2) = 4428300544
(7) id(obj1.data) = 4430915136
(8) id(obj2.data) = 4430915136

which is incorrect. In this case, when I created obj2, the obj1.data entry is lost. I would want the output in lines (2) and (3) to match but instead (3) and (4) are now the same. I can see down in (7) and (8) that the property is referencing the same location from both classes so I can see that the creation of obj2 is stomping on obj1.

I think I know what's going on but I wanted to confirm. I think the problem is in the assignment of the default value in my setattr(self, varname, default) line from typed_property. The default parameter to typed_property is really a reference to a single object in memory... So when that assignment happens what's really happening is a reference to default is being assigned to my property's internal storage, right?

I can fix the issue by changing setattr(self, varname, default) to be setattr(self, varname, copy.deepcopy(default) but is that the best way to go?

Ideally, I'd like to just use a dataclass but unfortunately it doesn't handle all the cases we need to cover.

If someone can articulate or post a link to some info that explains what's going on here in memory with that default function argument it'd be helpful for my understanding. Are arguments always fixed references or is it the default argument that's created once by Python?

It would be nice to either confirm what I'm thinking or learn more about the internals of the language.

Thanks!

python-dataclasses python-3.x python
1 Answers

You're correct that the reference to the default value is shared between MyClass instances, and your tests confirm as much. One important piece of information to understand why this happens is that, other than e.g. the body of an __init__ function, a class-body only gets evaluated once; on class creation. There is no way for two or more different default-objects to exist, because the code related to it does not get executed when creating a new MyClass instance.

dataclasses solve this issue of having default values defined in the class body by having both default for immutable default values and default_factory for mutable default values. I'd suggest to just use a similar pattern for your construct, the alternative of creating copies is bound to create problems in case you actually do want to share an object between instances:

def typed_property(name, default_factory=lambda: None):
    varname = "_" + name

    @property
    def prop(self):
        if not hasattr(self, varname):
            setattr(self, varname, default_factory())
        return getattr(self, varname)

    @prop.setter
    def setter(self, value):
        setattr(self, varname, value)

    return prop

class MyClass:
    data = typed_property("data", default_factory=dict)

By passing the dict function as the factory that gets called with the initial settatr, you get new dictionary objects for each instance. If you want to share some object o = MySharedObject(), just define the field as typed_property("shared_data", lambda: o). Or go all the way and define both default and default_factory arguments, but it will make the typed_property implementation a bit more complex with checking that only either the one or the other is used and what not.

And to show that it works now:

>>> a = MyClass()
>>> id(a.data)
140412230286208
>>> b = MyClass()
>>> id(b.data)
140412230275328
>>> id(a.data)
140412230286208
user_9621900
Related