When is `finally` run in a Python generator?

Question

I think I might misunderstand how the finally clause of a try/except/finally block works in Python, for generators.

In the following block of code, a generator starts a thread, and if the caller exits for any reason, the thread is cleaned up. That's the intention, at least.

However, I've noticed that in some strange situations, the finally block isn't run: If the caller raises and then catches their own exception, and assigns the exception object to a variable, the finally isn't run. I have no idea why this would be the case.

Here's the code.

import signal
from threading import Thread
import time


class MyThread(Thread):
    def __init__(self):
        super().__init__()
        self._stopped = False
    def run(self):
        while not self._stopped:
            time.sleep(0.2)
    def stop(self):
        self._stopped = True


class ThreadRunner:
    def start(self):
        self._my_thread = MyThread()
        self._my_thread.start()
    def end(self):
        self._my_thread.stop()
        self._my_thread.join()
        print('ThreadRunner end!')


def loop_forever():
    thread_runner = ThreadRunner()
    try:
        thread_runner.start()
        yield
    finally:
        print('loop_forever() is all done!')  # When does this line get run?
        thread_runner.end()


def listener():
    print('listener begin!')
    looper = loop_forever()
    next(looper)
    try:
        raise Exception()
    except Exception as e:
        es = e  # If you comment out this line (and replace it with `pass`), it doesn't hang
    print('listener... done!')


def main():
    def handle_exit(signum, *args): 
        raise Exception("SIGINT: EXITING")
    signal.signal(signal.SIGINT, handle_exit)
    listener()


if __name__ == "__main__":
    main()
    print('This program has exited. Or has it?')

What you'll see if run this code is this:

Press CTRL+C to trigger a signal handler. That will raise an exception, which triggers the finally in question.

As I said above, if remove es = e (replace it with pass), for some reason, our code exits as expected.

def listener():
    print('listener begin!')
    looper = loop_forever()
    next(looper)
    try:
        raise Exception()
    except Exception as e:
        pass
        # es = e  # If you comment out this line (and replace it with `pass`), it doesn't hang
    print('listener... done!')

Also, if I re-write listener() to use a for loop instead of next(), our code exits as expected:

def listener():
    print('listener_using_next')
    for _ in loop_forever():
        try:
            raise Exception()
        except Exception as e:
            es = e
        print('listener_using_next... done!')

Edit: In response to one comment, I want to note that our code exits as expected even if you return early, interrupting the generator.

def listener():
    print('listener_using_next')
    for _ in loop_forever():
        try:
            raise Exception()
        except Exception as e:
            es = e
            return
        print('listener_using_next... done!')

Finally, if I invoke garbage collection after main() (with import gc;gc.collect()), our code exits almost as expected: The "This program has exited" line prints, followed by "loop_forever() is all done!"

if __name__ == "__main__":
    main()
    print('This program has exited. Or has it?')
    import gc;gc.collect()

Can someone point to me to an explanation of how finally: works for generators? Ideally, in such a way that would help explain this strange behavior?

user_2357112 · Answer

finally in a generator is run under the same conditions as in any other code: when the execution of the try block finishes, as well as execution of any except block that got triggered, the finally runs. However, relative to most non-generator code, it is much easier in a generator for these conditions to just not happen.

When you suspend your generator in the middle of a try, the conditions for the finally to run have not occurred. If the generator just never resumes again, the conditions for the finally to run will never occur.

To try to bandage this a bit, when a generator is garbage collected, Python will throw a GeneratorExit exception into the generator. This usually causes the generator to run any pending finally blocks and finish up, but if the generator catches the exception and suspends again, or if the generator doesn't ever get garbage collected, finally blocks may not run.

In your test case, by saving the exception to the es variable, you create a reference cycle (through the exception object's stack trace) which keeps the generator alive. The es = e is necessary to create the reference cycle because Python sticks an implicit del e at the end of the except specifically to avoid a reference cycle. Python then just doesn't run garbage collection until the end of the program, at which point your generator finally gets cleaned up.

Note that there is no guarantee garbage collection will run, or that it will clean up everything it "should" clean up, and as you've seen, even if it does run, it doesn't have to run any time soon.

When you use a for loop instead of next, you're not leaving the generator suspended in the middle of the try. for runs the generator to completion, immediately and naturally running the finally.

When is `finally` run in a Python generator?

1 Answers