asyncio.gather() - task.cancelled() is False after task.cancel()

user_2889970 0
Viewed 960

After cancelling an asyncio.gather() task i would expect task.cancelled() to return True but instead it returns False.

Questions

  • Is this expected?
  • While Future.cancel() sets self._state = _CANCELLED the overridden _GatheringFuture.cancel() does not set self._state. (task.cancelled() uses task._state). Is this by design?

Example

def test_gather_cancellation_cancels_children_but_not_itself():

    # loop
    loop = new_event_loop()
    set_event_loop(loop)

    # create tasks and schedule them in gather
    task_child = ensure_future(sleep(1.0, result=1))
    task_gather = gather(task_child, return_exceptions=False)

    # assert nothing cancelled/done
    assert task_child.cancelled() is False
    assert task_gather.cancelled() is False
    assert task_child.done() is False
    assert task_gather.done() is False

    # cancel and await finishing of children
    assert task_gather.cancel() is True
    try:
        with pytest.raises(CancelledError):
            loop.run_until_complete(task_gather)
    finally:
        set_event_loop(None)
        loop.close()

    # assert all cancelled
    assert task_child.cancelled() is True

    # SURPRISING ASSERTION HERE:
    # surprisingly this is False because the the internal
    # task._state variable is not set in overridden
    # _GatheringFuture.cancel()
    assert task_gather.cancelled() is False  # this is surprising, expected True

    # assert all done
    assert task_child.done() is True
    assert task_gather.done() is True


============================= test session starts =============================
platform win32 -- Python 3.8.3rc1, pytest-5.4.2, py-1.8.1, pluggy-0.13.1 -- 
collecting ... collected 1 item

tests/gather/test_gather.py::test_gather_cancellation_cancels_children_but_not_itself PASSED [100%]

Docs

The Python docs mention this:

If gather() is cancelled, all submitted awaitables (that have not completed yet) are also cancelled. If any Task or Future from the aws sequence is cancelled, it is treated as if it raised CancelledError – the gather() call is not cancelled in this case. This is to prevent the cancellation of one submitted Task/Future to cause other Tasks/Futures to be cancelled.

However, it seems to me that this would apply only when gathered tasks are cancelled from outside! When running cancel() explicitly on the gather() task (and therefore its children), then it seems surprising that task.cancelled() is False. Therefore i'm not sure if the case described in the docs applies here.

async-await python-asyncio python-3.x python
3 Answers

I've also done a little digging and as mentioned initially and confirmed by @Aaron, there seems to be no way for an explicitly cancelled gather() future to get into a CANCELLED state, only FINISHED. This seems kind of unintuitive given the official future API of cancel() and cancelled(), so the question is:

  • Design choice, bug or accepted minor introspection inconvenience for this particular Future subclass. Which one is it? :)

Below is some more detail, from my digging:

What seems to happen

Here is another small snippet for testing and reproducing:

  • future_gather.cancel() cancels task_child
  • await future_gather will cause task_child to run and raise CancelledError
  • task_child's done callback for future_gather is invoked
  • _done_callback(task_child) calls future_gather.set_exception(CancelledError()) which sets:
    • future_gather._exception = CancelledError()
    • future_gather._state = _FINISHED
  • await calls future_gather.result() which raises self._exception if set (CancelledError in this case)

Result

Therefore the end result is a future instance whose:

  • .cancel() has been called
  • has explicitly raised a CancelledError to clients
  • returns a CancelledError for .exception()
  • ! returns False for .cancelled()
  • has a _state of FINISHED (not CANCELLED)

It seems kinda surprising that an object offering a cancel() method would always return False for cancelled().

user_28899700

For anyone interested: I have filed a bug report on bugs.python.org. The issue has been reviewed by asyncio core devs and is generally regarded as a lower priority bug. I also created a PR to fix it which awaits review on low priority.

user_28899700

I wrote a simplified script with the same logic as yours, more or less:

#! python3.8

import asyncio

async def main():
    # create tasks and schedule them in gather
    task_child = asyncio.create_task(asyncio.sleep(1.0))
    task_gather = asyncio.gather(task_child)

    print(task_gather.cancel())
    try:
        await task_gather
    except asyncio.CancelledError:
        pass
    print(task_child.cancelled(), task_child.done())
    print(task_gather.cancelled(), task_gather.done())

asyncio.run(main())

Output:

True
True True
False True

As you can see, task_gather.cancelled() also returned False in my case. I think the issue is that cancel() is called before task_gather is awaited (your program does the same thing). The docs are not clear, I think, about the returned value of cancelled() in that case. Neither task_child nor task_gather begins to execute until the await statement is encountered, so in effect you are cancelling a Future before it begins.

Note that "task_gather" is not really a Task but a Future.

If I remove the try:except: logic around await task_gather, the program aborts with a traceback and only the first print statement is seen.

user_24426130
Related