How can I integrate asyncio with an external event loop?

Viewed 459

I'm writing an event-driven program with the event scheduling written in C. The program uses Python for extension modules. I would like to allow extension modules to use async/await syntax to implement coroutines. Coroutines will just interact with parts of my program, no IO is involved. My C scheduler is single-threaded and I need coroutines to execute in its thread.

In a pure Python program, I would just use asyncio as is, and let my program use its event loop to drive all events. This is however not an option; my event loop needs to serve millions of C-based events per second and I cannot afford Python's overheads.

I tried to write my own event loop implementation, that delegates all scheduling to my C scheduler. I tried a few approaches:

  • Re-implement EventLoop, Future, Task etc to imitate how asyncio works (minus IO), so that call_soon delegates scheduling to my C event loop. This is safe, but requires a bit of work and my implementation will always be inferior to asyncio when it comes to documentation, debugging support, intricate semantic details, and correctness/test coverage.
  • I can use vanilla Task, Future etc from asyncio, and only create a custom implementation of AbstractEventLoop, delegating scheduling to my C event loop in the same way. This is pretty straightforward, but I can see that the vanilla EventLoop accesses non-obvious internals (task._source_traceback, _asyncgen_finalizer_hook, _set_running_loop), so my implementation is still second class. I also have to rely on the undocumented Handle._run to actually invoke callbacks.
  • Things appeared to get simpler and better if I subclassed from BaseEventLoop instead of AbstractEventLoop (but docs say I shouldn't do that). I still need Handle._run, though.
  • I could spawn a separate thread that run_forever:s a vanilla asyncio.DefaultEventLoop and run all my coroutines there, but coroutines depend on my program's extension API, which does not support concurrent calls. So I must somehow make DefaultEventLoop pause my C event loop while calling Handle._run(). I don't see a reasonable way to achieve that.

Any ideas on how to best do this? How did others solve this problem?

1 Answers

I found that trio, a third-party alternative to asyncio, provides explicit support for integration with alien event loops through something called guest mode. Solves my problem!

Related