When is the reference count for a local variable in a python function decreased?

Viewed 413

I have the following function:

def myfn():
    big_obj = BigObj()
    result = consume(big_obj)
    return result

When is the reference count for the value of BigObj() increased / decreased: Is it:

  1. when consume(big_obj) is called (since big_obj is not referenced afterwards in myfn)
  2. when the function returns
  3. some point, I don't no yet

Would it make a difference to change the last line to:

return consume(big_obj)

Edit (clarification for comments):

  • A local variable exists until the function returns
  • the reference can be deleted with del obj

But what is with temporaries (e.g f1(f2())?

I checked references to temporaries with this code:

import sys
def f2(c):
    print("f2: References to c", sys.getrefcount(c))


def f0():
    print("f0")
    f2(object())

def f1():
    c = object()
    print("f1: References to c", sys.getrefcount(c))
    f2(c)

f0()
f1()

This prints:

f0
f2: References to c 3
f1: References to c 2
f2: References to c 4

It seems, that references to temporary variables are held. Not that getrefcount gives one more than you would expect because it holds a reference, too.

3 Answers

When is the reference count for big_obj decreased

big_obj does not have a reference count. Variables don't have reference counts. Values do.

big_obj = BigObj()

This line of code creates an instance of the BigObj class. The reference count for that instance may increase or decrease multiple times, depending on the implementation details of that creation process (which is not necessarily written in Python). Notably, though, the assignment to the name big_obj increases the reference count.

when the function returns

At this point, the name big_obj ceases to exist - the name does not disappear simply because it won't be used again. (That's really hard to detect in the general case, and there isn't a particular benefit to it normally). If you must cause a name to cease to exist at a specific point in the operation (for example, because you know that is the last reference and want to trigger garbage collection; or perhaps because you are doing something tricky with __weakref__) then that is what the del statement is for.

Because a name for the object ceases to exist, its reference count decreases. If and when that count reaches zero, the object is garbage collected. It may have any number of references stored in other places, for a wide variety of reasons. (For example, there could be a bug in C code that implements the class; or the class might deliberately maintain its own list of every instance ever created.)


Note that all of the above pertains specifically to the reference implementation. In other implementations, things will differ. There might be some other trigger for garbage collection to happen. There might not be reference counting at all (as with Jython).

From the comments, it seems like what you are worried about is the potential for a memory leak. The code that you show cannot cause a memory leak - but neither can it fix a memory leak caused elsewhere. In Python, as in garbage-collected languages in general, memory leaks happen because objects hold on to references to each other that aren't needed. But there is no concept of "ownership" or "transfer" of references normally - all you need to do is not do things like "maintain a list of every instance ever created" without a) a good reason and b) a way to take instances out of that list when you want to forget about them.

A local variable, though, by definition, cannot extend the lifetime of an object beyond the local scope.

Disclaimer: Most information is from the comments. So credit for every one who participated in the discussion.

When an object is deleted is an implementation detail in general. I will refer to CPython, which is based on reference counting. I ran the code examples with CPython 3.10.0.

  • An object is deleted, when the reference count hits zero.
  • Returning from a function deletes all local references.
  • Assigning a name to a new value decreases the reference count of the old value
  • passing a local increases the reference count. The reference is in on the stack(frame)
  • Returning from a function removes the reference from the stack

The last point is even valid for temporary references like f(g()). The last reference to g() is deleted, when f returns (assuming that g does not save a reference somewhere)see here

So for the example from the question:

def myfn():
    big_obj = BigObj() # reference 1                     
    result = consume(big_obj) # reference 2 on the stack frame for  
                              # consume. Not yet counting any 
                              # reference inside of consume
                              # after consume returns: The stack frame 
                              # and reference 2 are deleted. Reference  
                              # 1 remains
    return result             # myfn returns reference 1 is deleted. 
                              # BigObj is deleted
def consume(big_obj):
    pass # consume is holding reference 3

If we would change this to:

def myfn():
    return consume(BigObj()) # reference is still saved on the stack 
                             # frame and will be only deleted after  
                             # consume returns
def consume(big_obj):
    pass # consume is holding reference 2

How can I check reliably, if an object was deleted?

You cannot rely on gc.get_objects(). gc is used to detect and recycle reference cycles. Not every reference is tracked by the gc. You can create a weak reference and check if the reference is still valid.

class BigObj:
    pass

import weakref
ref = None

def make_ref(obj):
    global ref
    ref = weakref.ref(obj)
    return obj

def myfn():
    return consume(make_ref(BigObj()))

def consume(obj):
    obj = None # remove to see impact on ref count
    print(sys.getrefcount(ref()))
    print(ref()) # There is still a valid reference. It is the one from consume stack frame

myfn()

How to pass a reference to a function and remove all references in the calling function?

You can box the reference, pass to the function and clear the boxed reference from inside the function:

class Ref:
    def __init__(ref):
        self.ref = ref
    def clear():
        self.ref = None

def f1(ref):
    r = ref.ref
    ref.clear()

def f2():
    f1(Ref(object())

Variables have function scope in Python, so they aren't removed until the function returns. As far as I can tell, you can't destroy a reference to a local variable in a function from outside that function. I added some gc calls in the example code to test this.

import gc
class BigObj:
    pass
def consume(obj):
    del obj  # only deletes the local reference to obj, but another one still exists in the calling function

def myfn():
    big_obj = BigObj()
    big_obj_id = id(big_obj)  # in CPython, this is the memory address of the object
    consume(big_obj)
    print(any(id(obj) == big_obj_id for obj in gc.get_objects()))
    return big_obj_id

>>> big_obj_id = myfn()
True
>>> gc.collect()  # I'm not sure where the reference cycle is, but I needed to run this to clean out the big object from the gc's list of objects in my shell
>>> print(any(id(obj) == big_obj_id for obj in gc.get_objects()))
False

Since True was printed, the big object still existed after we forced garbage collection to occur even though there were no references to that variable after that point in the function. Forcing garbage collection after the function returns rightfully determines that the reference count to the big object is 0, so it cleans that object up. NOTE: As the comments below point out, ids for deleted objects may be reused so checking for equal ids may result in false positives. However, I'm confident that the conclusion is still correct.

One thing you can do to reclaim that memory earlier is to make the big object global, which could allow you to delete it from within the called function.

def consume():
    # do whatever you need to do with the big object
    big_obj_id = id(big_obj)
    del globals()["big_obj"]
    print(any(id(obj) == big_obj_id for obj in gc.get_objects()))
    # do anything else you need to do without the big object

def myfn():
    globals()["big_obj"] = BigObj()
    result = consume()
    return result

>>> myfn()
False

This sort of pattern is pretty weird and likely very hard to maintain though, so I would advise against using this. If you only need to delete the big object right after consume() is called, you could do something like this in order to free up the memory used by the big object as soon as possible.

big_obj = BigObj()
consume(big_obj)
del big_obj

Another strategy you might try is deleting the references within the big object that's passed in from the consume() function with del big_obj.x for some attribute x.

Related