Python destructor called in the "wrong" order based on reference count

Viewed 1065

As far as I understand Python destructors should be called when the reference count of an object reaches 0. But this assumption seems not to be correct. Look at the following code:

class A:
    def __init__(self, b):
        self.__b = b
        print("Construct A")
        
    def __del__(self):
        # It seems that the destructor of B is called here.
        print("Delete A")
        # But it should be called here
        
class B:
    def __init__(self):
        print("Construct B")
        
    def __del__(self):
        print("Delete B")
        
b = B()
a = A(b)

Outputs

Construct B                                                                                                                                                                                                                                   
Construct A                                                                                                                                                                                                                                   
Delete B                                                                                                                                                                                                                                      
Delete A

But A has a reference to B, so I would expect the following output:

Construct B                                                                                                                                                                                                                                   
Construct A                                                                                                                                                                                                                                   
Delete A                                                                                                                                                                                                                                      
Delete B

What am I not getting?

4 Answers

So, since the objects are still alive when the interpreter shuts down, you are actually not even guaranteed that __del__ will be called. At this point, the language makes no guarantees about when the finalizer is called.

From the docs:

It is not guaranteed that __del__() methods are called for objects that still exist when the interpreter exits.

Note, if you change the script to:

(py38) 173-11-109-137-SFBA:~ juan$ cat test.py
class A:
    def __init__(self, b):
        self.__b = b
        print("Construct A")

    def __del__(self):
        # It seems that the destructor of B is called here.
        print("Delete A")
        # But it should be called here

class B:
    def __init__(self):
        print("Construct B")

    def __del__(self):
        print("Delete B")
b = B()
a = A(b)

del a
del b

Then, executed:

(py38) 173-11-109-137-SFBA:~ juan$ python test.py
Construct B
Construct A
Delete A
Delete B

Although del does not delete objects, it deletes references, so it forces the reference count to reach 0 while the interpreter is still running, so the order is as you would expect.

Sometimes, __del__ won't be called at all. A common circumstance is file-objects created by

f = open('test.txt')

That have live references in the global scope. If not closed explicitly, it might not call __del__ and the file will not flush and you won't get anything written. Which is a great reason to use the file object as a context-manager...

Per the comments elsewhere on this question, you probably don't want to use __del__; it's not really a destructor in the C++ sense. You probably want to make the objects into context managers (by writing __enter__ and __exit__ methods) and use them in the with statement, and/or give them close methods which need to be called explicitly.

However, to answer the question as given: the reason is that both objects have references from the global variables a and b; neither reference count ever goes to zero. The destructor is called at the end when the python interpreter is shutting down and all the non-zero-count objects are being collected.

To see the behaviour you expect, put the a and b variables in a function so that the reference counts go to zero during the main part of execution.

class A:
    def __init__(self, b):
        self.__b = b
        print("Construct A")

    def __del__(self):
        print("Delete A")

class B:
    def __init__(self):
        print("Construct B")

    def __del__(self):
        print("Delete B")

def foo():
    b = B()
    a = A(b)

foo()

Among the things you're missing, there's a reference cycle. It goes roughly a->b->B->B.__init__->B.__init__.__globals__->a:

  • Your A instance has a reference to its __dict__, which has a reference to your B instance.
  • Your B instance has a reference to your B class.
  • Your B class has a reference to its __dict__, which has references to all of B's methods. (Technically, if you try to access B.__dict__ yourself, you'll get a mappingproxy wrapping B's "real __dict__". B has a reference to the real dict, not the proxy.)
  • B's methods each have a reference to their global variable dict.
  • The global variable dict has a reference to your A instance (because this dict is where the a global variable is).

When reclaiming objects in a reference cycle, there are no guarantees as to what order __del__ methods are executed in.

If you don't believe the reference cycle exists, it's fairly straightforward to demonstrate the existence of these references:

import gc

print(a.__dict__ in gc.get_referents(a))
print(b in gc.get_referents(a.__dict__))
print(B in gc.get_referents(b))
# this bypasses the mappingproxy
# never use this to modify a class's dict - you'll cause memory corruption
real_dict = next(d for d in gc.get_referents(B) if isinstance(d, dict))
print(B.__init__ in gc.get_referents(real_dict))
print(B.__init__.__globals__ in gc.get_referents(B.__init__))
print(a in gc.get_referents(B.__init__.__globals__))

All of these prints print True.


Aside from that, there are a few relevant points other answers have already brought up. Your objects survive to interpreter shutdown, so there is no guarantee that __del__ will be called at all. Also, __del__ is a finalizer, not a destructor. It doesn't have anywhere near the same kind of guarantees that an actual destructor would have in a language like C++.

This concept is called composition:

In composition, if the parent object is destroyed, then the child objects also cease to exist. Composition is actually a strong type of aggregation and is sometimes referred to as a “death” relationship. As an example, a house may be composed of one or more rooms. If the house is destroyed, then all of the rooms that are part of the house are also destroyed.

For an object to be deleted, all it's attributes need to be deleted first. From the above example, rooms of the house will be destroyed first in order for the house to be destroyed.

Therefore, A has an attribute of object B which will be deleted first.

Related