Understanding __get__ and __set__ and Python descriptors

Viewed 170321

I am trying to understand what Python's descriptors are and what they are useful for. I understand how they work, but here are my doubts. Consider the following code:

class Celsius(object):
    def __init__(self, value=0.0):
        self.value = float(value)
    def __get__(self, instance, owner):
        return self.value
    def __set__(self, instance, value):
        self.value = float(value)


class Temperature(object):
    celsius = Celsius()
  1. Why do I need the descriptor class?

  2. What is instance and owner here? (in __get__). What is the purpose of these parameters?

  3. How would I call/use this example?

8 Answers

Before going into the details of descriptors it may be important to know how attribute lookup in Python works. This assumes that the class has no metaclass and that it uses the default implementation of __getattribute__ (both can be used to "customize" the behavior).

The best illustration of attribute lookup (in Python 3.x or for new-style classes in Python 2.x) in this case is from Understanding Python metaclasses (ionel's codelog). The image uses : as substitute for "non-customizable attribute lookup".

This represents the lookup of an attribute foobar on an instance of Class:

enter image description here

Two conditions are important here:

  • If the class of instance has an entry for the attribute name and it has __get__ and __set__.
  • If the instance has no entry for the attribute name but the class has one and it has __get__.

That's where descriptors come into it:

  • Data descriptors which have both __get__ and __set__.
  • Non-data descriptors which only have __get__.

In both cases the returned value goes through __get__ called with the instance as first argument and the class as second argument.

The lookup is even more complicated for class attribute lookup (see for example Class attribute lookup (in the above mentioned blog)).

Let's move to your specific questions:

Why do I need the descriptor class?

In most cases you don't need to write descriptor classes! However you're probably a very regular end user. For example functions. Functions are descriptors, that's how functions can be used as methods with self implicitly passed as first argument.

def test_function(self):
    return self

class TestClass(object):
    def test_method(self):
        ...

If you look up test_method on an instance you'll get back a "bound method":

>>> instance = TestClass()
>>> instance.test_method
<bound method TestClass.test_method of <__main__.TestClass object at ...>>

Similarly you could also bind a function by invoking its __get__ method manually (not really recommended, just for illustrative purposes):

>>> test_function.__get__(instance, TestClass)
<bound method test_function of <__main__.TestClass object at ...>>

You can even call this "self-bound method":

>>> test_function.__get__(instance, TestClass)()
<__main__.TestClass at ...>

Note that I did not provide any arguments and the function did return the instance I had bound!

Functions are Non-data descriptors!

Some built-in examples of a data-descriptor would be property. Neglecting getter, setter, and deleter the property descriptor is (from Descriptor HowTo Guide "Properties"):

class Property(object):
    def __init__(self, fget=None, fset=None, fdel=None, doc=None):
        self.fget = fget
        self.fset = fset
        self.fdel = fdel
        if doc is None and fget is not None:
            doc = fget.__doc__
        self.__doc__ = doc

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        if self.fget is None:
            raise AttributeError("unreadable attribute")
        return self.fget(obj)

    def __set__(self, obj, value):
        if self.fset is None:
            raise AttributeError("can't set attribute")
        self.fset(obj, value)

    def __delete__(self, obj):
        if self.fdel is None:
            raise AttributeError("can't delete attribute")
        self.fdel(obj)

Since it's a data descriptor it's invoked whenever you look up the "name" of the property and it simply delegates to the functions decorated with @property, @name.setter, and @name.deleter (if present).

There are several other descriptors in the standard library, for example staticmethod, classmethod.

The point of descriptors is easy (although you rarely need them): Abstract common code for attribute access. property is an abstraction for instance variable access, function provides an abstraction for methods, staticmethod provides an abstraction for methods that don't need instance access and classmethod provides an abstraction for methods that need class access rather than instance access (this is a bit simplified).

Another example would be a class property.

One fun example (using __set_name__ from Python 3.6) could also be a property that only allows a specific type:

class TypedProperty(object):
    __slots__ = ('_name', '_type')
    def __init__(self, typ):
        self._type = typ

    def __get__(self, instance, klass=None):
        if instance is None:
            return self
        return instance.__dict__[self._name]

    def __set__(self, instance, value):
        if not isinstance(value, self._type):
            raise TypeError(f"Expected class {self._type}, got {type(value)}")
        instance.__dict__[self._name] = value

    def __delete__(self, instance):
        del instance.__dict__[self._name]

    def __set_name__(self, klass, name):
        self._name = name

Then you can use the descriptor in a class:

class Test(object):
    int_prop = TypedProperty(int)

And playing a bit with it:

>>> t = Test()
>>> t.int_prop = 10
>>> t.int_prop
10

>>> t.int_prop = 20.0
TypeError: Expected class <class 'int'>, got <class 'float'>

Or a "lazy property":

class LazyProperty(object):
    __slots__ = ('_fget', '_name')
    def __init__(self, fget):
        self._fget = fget

    def __get__(self, instance, klass=None):
        if instance is None:
            return self
        try:
            return instance.__dict__[self._name]
        except KeyError:
            value = self._fget(instance)
            instance.__dict__[self._name] = value
            return value

    def __set_name__(self, klass, name):
        self._name = name

class Test(object):
    @LazyProperty
    def lazy(self):
        print('calculating')
        return 10

>>> t = Test()
>>> t.lazy
calculating
10
>>> t.lazy
10

These are cases where moving the logic into a common descriptor might make sense, however one could also solve them (but maybe with repeating some code) with other means.

What is instance and owner here? (in __get__). What is the purpose of these parameters?

It depends on how you look up the attribute. If you look up the attribute on an instance then:

  • the second argument is the instance on which you look up the attribute
  • the third argument is the class of the instance

In case you look up the attribute on the class (assuming the descriptor is defined on the class):

  • the second argument is None
  • the third argument is the class where you look up the attribute

So basically the third argument is necessary if you want to customize the behavior when you do class-level look-up (because the instance is None).

How would I call/use this example?

Your example is basically a property that only allows values that can be converted to float and that is shared between all instances of the class (and on the class - although one can only use "read" access on the class otherwise you would replace the descriptor instance):

>>> t1 = Temperature()
>>> t2 = Temperature()

>>> t1.celsius = 20   # setting it on one instance
>>> t2.celsius        # looking it up on another instance
20.0

>>> Temperature.celsius  # looking it up on the class
20.0

That's why descriptors generally use the second argument (instance) to store the value to avoid sharing it. However in some cases sharing a value between instances might be desired (although I cannot think of a scenario at this moment). However it makes practically no sense for a celsius property on a temperature class... except maybe as purely academic exercise.

You'd see https://docs.python.org/3/howto/descriptor.html#properties

class Property(object):
    "Emulate PyProperty_Type() in Objects/descrobject.c"

    def __init__(self, fget=None, fset=None, fdel=None, doc=None):
        self.fget = fget
        self.fset = fset
        self.fdel = fdel
        if doc is None and fget is not None:
            doc = fget.__doc__
        self.__doc__ = doc

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        if self.fget is None:
            raise AttributeError("unreadable attribute")
        return self.fget(obj)

    def __set__(self, obj, value):
        if self.fset is None:
            raise AttributeError("can't set attribute")
        self.fset(obj, value)

    def __delete__(self, obj):
        if self.fdel is None:
            raise AttributeError("can't delete attribute")
        self.fdel(obj)

    def getter(self, fget):
        return type(self)(fget, self.fset, self.fdel, self.__doc__)

    def setter(self, fset):
        return type(self)(self.fget, fset, self.fdel, self.__doc__)

    def deleter(self, fdel):
        return type(self)(self.fget, self.fset, fdel, self.__doc__)

Easy to digest (with example) Explanation for __get__ & __set__ & __call__ in classes, what is Owner, Instance?

Some points to mug up before diving in:

  1. __get__ __set__ are called descriptors of the class to work/save their internal attributes namely: __name__ (name of class/owner class), variables - __dict__ etc. I will explain what is an owner later
  2. Descriptors are used in design patterers more commonly, for example, with decorators (to abstract things out). You can consider it's more often used in software architecture design to make things less redundant and more readable (seems ironical). Thus abiding SOLID and DRY principles.
  3. If you are not designing software that should abide by SOLID and DRY principles, you probably don't need them, but it's always wise to understand them.

1. Conside this code:

class Method:
    def __init__(self, name):
        self.name = name
    def __call__(self, instance, arg1, arg2):
        print(f"{self.name}: {instance} called with {arg1} and {arg2}")


class MyClass:
    method = Method("Internal call")

instance = MyClass()


instance.method("first", "second")

# Prints:TypeError: __call__() missing 1 required positional argument: 'arg2'

So, when instance.method("first", "second") is called, __call__ method is called from the Method class (call method makes a class object just callable like a function - whenever a class instance is called __call__ gets instiantiated), and following arguments are assigned: instance: "first", arg1: "second", and the last arg2 is left out, this prints out the error: TypeError: __call__() missing 1 required positional argument: 'arg2'

2. how to solve it?

  • Since __call__ takes instance as first argument (instance, arg1, arg2), but instance of what?

  • Instance is the instance of main class (MyClass) which is calling the descriptor class (Method). So, instance = MyClass() is the instance and so who is the owner? the class holding the discriptor class - MyClass, However, there is no method in our descriptor class (Method) to recognise it as an instance. So that is where we need __get__ method. Again consider the code below:



from types import MethodType
class Method:
    def __init__(self, name):
        self.name = name
    def __call__(self, instance, arg1, arg2):
        print(f"{self.name}: {instance} called with {arg1} and {arg2}")
    def __set__(self, instance, value):
        self.value = value
        instance.__dict__["method"] = value
    def __get__(self, instance, owner):
        if instance is None:
            return self
        print (instance, owner)
        return MethodType(self, instance)   


class MyClass:
    method = Method("Internal call")

instance = MyClass()


instance.method("first", "second") 
# Prints: Internal call: <__main__.MyClass object at 0x7fb7dd989690> called with first and second

forget about set for now according to docs:

__get__ "Called to get the attribute of the owner class (class attribute access) or of an instance of that class (instance attribute access)."

if you do: instance.method.__get__(instance)

Prints:<__main__.MyClass object at 0x7fb7dd9eab90> <class '__main__.MyClass'>

this means instance: object of MyClass which is instance and Owner is MyClass itself

3. __set__ Explaination:

__set__ is used to set some value in the class __dict__ object (let's say using a command line). command for setting the internal value for set is: instance.descriptor = 'value' # where descriptor is method in this case

  • (instance.__dict__["method"] = value in the code just update the __dict__ object of the descriptor)

  • So do: instance.method = 'value' now to check if the value = 'value' is set in the __set__ method we can access __dict__ object of the descriptor method. Do: instance.method.__dict__ prints: {'_name': 'Internal call', 'value': 'value'}

  • Or you can check the __dict__ value using vars(instance.method) prints: {'name': 'Internal call', 'value': 'value'} I hope things are clear now:)

Related