Python - asyncio - Task exception was never retrieved

user_10064174 0
Viewed 7748

Description: (simplified)

  • I have 2 tasks.
  • Within each task I have 3 coroutines.
  • 2 coroutines from the first task fail. (simulated)
  • While processing task results, I am getting one "Task exception was never retrieved" message.
  • I believe this is because exception of only one of the two failed coroutines in that task was processed.
  • How do I process exceptions of both coroutines within task and/or avoid the "Task exception was never retrieved" message?

Code: (simplified)

import asyncio
async def download(data):
    filename = "*" if data in ["b", "c"] else data  # simulated failure
    with open(filename, "w") as f:
        f.write(data)
async def coro(data_list):
    coroutines = [download(data) for data in data_list]
    for coroutine in asyncio.as_completed(coroutines):
        await coroutine
async def main():
    task1 = asyncio.create_task(coro(["a", "b", "c"]))
    task2 = asyncio.create_task(coro(["d", "e", "f"]))
    results = await asyncio.gather(task1, task2, return_exceptions=True)
    for _ in results:
        pass
asyncio.run(main())

Output: (simplified)

Task exception was never retrieved
future: <Task finished coro=<download() done, defined at D:/myscript.py:2> exception=OSError(22, 'Invalid argument')>
Traceback (most recent call last):
  File "D:/myscript.py", line 4, in download
    with open(filename, "w") as f:
OSError: [Errno 22] Invalid argument: '*'
python-asyncio coroutine asynchronous python
1 Answers

If you want to collect exceptions instead of raising them, you can use asyncio.gather(return_exceptions=True) in coro as well. For example:

import asyncio

async def download(data):
    if data in ['b', 'c']:
        1/0    # simulate error
    return 42  # success

async def coro(data_list):
    coroutines = [download(data) for data in data_list]
    return await asyncio.gather(*coroutines, return_exceptions=True)

async def main():
    task1 = asyncio.create_task(coro(["a", "b", "c"]))
    task2 = asyncio.create_task(coro(["d", "e", "f"]))
    return await asyncio.gather(task1, task2, return_exceptions=True)

print(asyncio.run(main()))

This will print:

[[42, ZeroDivisionError('division by zero'), ZeroDivisionError('division by zero')], [42, 42, 42]]
user_16008980
Related