Mixed comprehension (async + sync) does not work as expected in Python

user_3105524 0
Viewed 203

I am trying to make my code better by omitting mutable list and replacing comprehension. There is a nested 'for' iterator which the inner is async. Here is the working code:

async def get_accumulator_providers(order: OrderDTO) -> Tuple[str, ...]:
    bucket_mapping = await read_config(order)
    result = []
    for bucket in bucket_mapping:
        async for item in dispatch_bucket(order, bucket, bucket_mapping[bucket], frozenset(accumulator_filters)):
            result.append(item)
    return tuple(result)

But when I use this code instead:

async def get_accumulator_providers(order: OrderDTO) -> Tuple[str, ...]:
    bucket_mapping = await read_config(order)

    return tuple(item for bucket in bucket_mapping async for item in
             dispatch_bucket(order, bucket, bucket_mapping[bucket], frozenset(accumulator_filters)))

I get this error:

 'async_generator' object is not iterable

What is the problem with my nested comprehension?

async-await python-asyncio list-comprehension iterator python
2 Answers

The problem here is that tuple uses synchronous iterator protocol.

Yuru Selivanov answered here:

... result = list(await g(i) for i in range(3))

This is equivalent to this code:

 async def ait():
     for i in range(3):
         v = await g(i)
         yield v

 result = list(ait())

Where ait is an async generator function. You can't iterate it with the regular for x in ... syntax, and you can't pass it to functions that expect a synchronous iterator (such as list).

Similarly, with synchronous code:

 a = (i for i in range(3))
 a[0]
 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
 TypeError: 'generator' object is not subscriptable

where (for ...) is another syntax for defining a synchronous generator.

... result = [await g(i) for i in range(3)]

This is equivalent to this code:

 result = []
 for i in range(3):
     v = await g(i)
     result.append(v)

I agree that PEP 530 is a bit vague about this and can be updated. I'll take a look into that.

Perhaps we can make the "TypeError: 'async_generator' object is not iterable" error message a bit clearer. Any ideas to improve it are welcome.

I would say that the first case should either behave as a second one, or raise a syntax error.

No, but we can improve error messages.

user_137826690

As mentioned @VisioN and I found here, the problem is not mixing async and sync for loops. The problem is using tuple() for joining iterations. Using a [] as list comprehension fixed the problem. Actually, tuple() is not list comprehension but just gets a generator and calls 'iter' for making a tuple. Therefore, the following code works:

async def get_accumulator_providers(order: OrderDTO) -> Tuple[str, ...]:
bucket_mapping = await read_config(order)

return tuple([item for bucket in bucket_mapping async for item in
         dispatch_bucket(order, bucket, bucket_mapping[bucket], frozenset(accumulator_filters))])
user_31055240
Related