How to run coroutine in a ThreadPoolExecutor?

Viewed 35

I have 2 functions:

  1. A method that maintains a streaming RPC connection (with a third party service) that constantly receives data in a loop.
  2. A method that periodically checks for the data received and performs some computation.

e.g.,

def streaming_loop(global_shared_data, data_lock: asyncio.Lock):
  # A generator for input.
  input_generator = input.generator()
  # stream_data is a third party API that takes in a generator and maintains a 
  # RPC connection 
  responses_generator = client.stream_data(input_generator)
  for response in responses_generator:
     # Write data in a thread safe manner
     await write_data(response, global_shared_data, data_lock)

async def process_loop(global_shared_data, data_lock: asyncio.Lock):
   while True:
      # To schedule other tasks in between.
      asyncio.sleep(0)
      async with self._lock:
         # process data in a thread safe manner
         await process_data(global_shared_data, data_lock)

I want to run streaming_loop in a ThreadPoolExecutor since I cannot make it a co-routine due to the third party API client.stream_data consumes input as a generator. I also want to run process_loop as an asyncio task. However, when I tried

executor = ThreadPoolExecutor(max_workers=2)
loop = asyncio.get_running_loop()
loop.run_in_executor(executor, streaming_loop(global_shared_data, lock))

I'm getting the error:

TypeError: 'coroutine' object is not callable

Is there no way to run a co-routine in a threadpoolexecutor? If that's not possible, how would I make this thread safe when the 2 methods are modifying the same global object?

0 Answers
Related