asyncio throws runtime error with exception ignored

user_7978112 0
Viewed 1130

Below is a simple program to collect URL lengths.

import aiohttp
import asyncio
from time import perf_counter


URLS = ['http://www.cnn.com', 'http://www.huffpost.com', 'http://europe.wsj.com',
        'http://www.bbc.co.uk', 'http://failfailfail.com']

async def async_load_url(url, session):
    try:
        async with session.get(url) as resp:
            content = await resp.read()
        print(f"{url!r} is {len(content)} bytes")
    except IOError:
        print(f"failed to load {url}")        

async def main():

    async with aiohttp.ClientSession() as session:
        tasks = [async_load_url(url, session) for url in URLS]
        await asyncio.wait(tasks)

if __name__ == "__main__":
    start = perf_counter()
    asyncio.run(main())
    elapsed = perf_counter() - start
    print(f"\nTook {elapsed} seconds")

Why is the following code failing with a runtime error with exception ignored in python 3.9? How to fix it?

Traceback is: RuntimeError: Event loop is closed specifically with Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x000001F8A7A713A0>

aiohttp python-asyncio python-3.x
2 Answers

This is caused by a known issue in aiohttp on Windows, for details check out the bug at https://github.com/aio-libs/aiohttp/issues/4324

There are a couple of hacks that silence this error. The first way is to get the event loop and call run_until_complete instead of asyncio.run(main()) like so:

asyncio.get_event_loop().run_until_complete(main())

Alternatively, changing the event loop policy to WindowsSelectorEventLoopPolicy before calling asyncio.run(main()) also works as the problem seems to occur when using WindowsProtractorEventLoopPolicy.

asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
asyncio.run(main())

Of course, the second solution will make your code platform specific, so be careful.

user_29173260

You are using asyncio.await to run all your tasks, but you are not checking the completed tasks for exceptions. await returns two sequences: one with the completed tasks, one with the pending tasks - you have to query the done tasks for exceptions:

async def main():
    async with aiohttp.ClientSession() as session:
        tasks = [asyncio.task(async_load_url(url, session), name=url) for url in URLS]
        done, pending = await asyncio.wait(tasks)
        for task in done:
            try:
                task.exception() # in task execption is re-raised
            except Exception as exc:
                print (f"Exception loading url {task.name}:\n {exc}")

If it is a long process and you want to handle exceptions as they happen, asyncio.wait offers an interface that facilitates that - just tell it when it should return:

        
    async def main():
    async with aiohttp.ClientSession() as session:
        tasks = [asyncio.task(async_load_url(url, session), name=url) for url in URLS]
        while tasks:
            done, tasks = await asyncio.wait(tasks, return_when=asyncio.FIRST_EXCEPTION)
            for task in done:
                try:
                    task.exception() # in task execption is re-raised
                except Exception as exc:
                    print (f"Exception loading url {task.name}:\n {exc}")
user_1082050
Related