asyncio gather yielding results as they come in

user_1722434 0
Viewed 1093

I want to be able to yield results from a set of tasks run by gather as they come in for further processing.

# Not real code, but an example

async for response in asyncio.gather(*[aiohttp.get(url) for url in ['https://www.google.com', 'https://www.amazon.com']]):
    await process_response(response)

At present, I can use the gather method to run all get concurrently, but must wait until they're all complete to process them. I'm still new to Python async/await, so maybe there's some obvious way of doing this I'm missing.

# What I can do now
responses = await asyncio.gather(*[aiohttp.get(url) for url in ['https://www.google.com', 'https://www.amazon.com']])
await asyncio.gather(*[process_response(response) for response in responses])

Thanks!

python-asyncio python
1 Answers

gather as you already noted will wait until all coroutines are done, thus you need to find another way.

For example you can use function asyncio.as_completed that seems to do exactly what you want.

import asyncio


async def echo(t):
    await asyncio.sleep(t)
    return t



async def main():
    coros = [
        echo(3),
        echo(2),
        echo(1),
    ]

    for first_completed in asyncio.as_completed(coros):
        res = await first_completed
        print(f'Done {res}')


asyncio.run(main())

Result:

Done 1
Done 2
Done 3
[Finished in 3 sec]
user_11132070
Related