Consuming events from multiple Azure Event Hubs in one Python program

user_17142364 0
Viewed 167

I have two event hubs, each with its own consumer group. I would like to receive events from each of them at the same time but I don't see anything related to this in the GitHub samples.

More specifically, I would like to take advantage of asyncio for reading from the two topics concurrently but I'm new to asynchronous programming and I can't figure it out. I tried something like this but it never moves on to reading with client2:

async with client1:
    async with client2:
        await client1.receive_batch(...)
        await client2.receive_batch(...)

I also tried separating the async with statements and even tried adding await asyncio.sleep(1) in the on_event_batch function, just to make sure that control is given back to the event loop.

At this point I'm not sure whether what I want to do simply cannot be achieved with asyncio or I just don't know how to use it.

Any help will be appreciated, thanks!

azure-eventhub python-asyncio azure python
1 Answers

receive_batch is designed to be a blocking call which keeps pumping events from the event hub infinitely.

To facilitate asyncio for coroutine tasks execution concurrently, you could use asyncio.create_task to execute receive_batch.

Example code:

async with client1, client2:
    future1 = asyncio.create_task(client1.receive_batch(...))
    future2 = asyncio.create_task(client2.receive_batch(...))
    results = await asyncio.gather(future1, future2)  # program will be blocking here waiting for two coroutines to finish

For more advanced usage of asyncio module, I would recommend you to take a look at the official Python docs for Coroutines and Tasks.

Please let me know if there's any more I could help on.

user_146095470
Related