Long (recursive) tracebacks when passing an error with future.set_exception(err)

user_947418 0
Viewed 185

I am having trouble understanding how python's traceback works together with asyncio. I have a function that accepts requests to be processed later in a batch. The error may occur while processing the batch and I use future.set_exception(err) to communicate the error to all callers. For some reason, it generates ridiculously long tracebacks (depends on a batch length).

Here is a minimum code to reproduce it:

import asyncio

batch = []
batch_len = 10

async def process():
    loop = asyncio.get_running_loop()
    future = loop.create_future()
    batch.append(future)
    if len(batch) >= batch_len:
        await process_batch()
    return await future

async def process_batch():
    try:
        # Here I call some outer service. Let's pretend it raises an error.
        raise RuntimeError("Oops")
    except Exception as err:
        for future in batch:
            # Pass an error to all callers
            future.set_exception(err)

async def main():
    futures = (process() for _ in range(batch_len))
    await asyncio.gather(*futures) # raises on first unsuccessful request

asyncio.run(main())

The output: (problematic part marked)

Traceback (most recent call last):
  File ".../fut1.py", line 27, in <module>
    asyncio.run(main())
  File ".../python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File ".../python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File ".../fut1.py", line 25, in main
    await asyncio.gather(*futures) # raises on first unsuccessful request
  File ".../fut1.py", line 12, in process  \
    return await future                    |
  File ".../fut1.py", line 12, in process  |
    return await future                    |  HERE IT IS
  File ".../fut1.py", line 12, in process  |
    return await future                    |
  [Previous line repeated 7 more times]    /
  File ".../fut1.py", line 17, in process_batch
    raise RuntimeError("Oops")
RuntimeError: Oops

Not sure why this recursive part appears here and how to avoid it. I will be grateful for any ideas.

python-asyncio python
1 Answers

I think this unexpected behaviour stems from a subtle mistake in process. You are not returning the future, but the result of the future. Meaning your coroutines will yield on the await future part, waiting for a result, before the resulting coroutines can be collected by asyncio.gather. This way the RuntimeError is actually raised within process when you are unpacking your generator and not just set as a result for the collected coroutines in gather. This is why you get 10 times the same exception in the same line of code in your traceback. Maybe this slightly changed piece of code leads to a more expected behaviour:

import asyncio

batch = []
batch_len = 10

async def process():
    loop = asyncio.get_running_loop()
    future = loop.create_future()
    batch.append(future)
    if len(batch) >= batch_len:
        await process_batch()
    # return the future without awaiting, otherwise exception is raised here
    return future 

async def process_batch():
    try:
        # Here I call some outer service. Let's pretend it raises an error.
        raise RuntimeError("Oops")
    except Exception as err:
        for future in batch:
            # Pass an error to all callers
            future.set_exception(err)

async def main():
    futures = (process() for _ in range(batch_len))
    results = await asyncio.gather(*futures) # raises on first unsuccessful request
    print(*results, sep="\n")

asyncio.run(main())
user_144121900
Related