Python 2 Tornado asychronous method

user_4178623 0
Viewed 163

I have to use Python 2 to make asynchronous calls.

I'm using Tornado to accomplish this but if there is a better tool I can change tools.

The following code runs my sleep 2 && echo hi command asynchronously in the background

from tornado import gen
import subprocess
import time

@gen.coroutine
def longProcess():
    bashCommand = "sleep 5 && echo hi"
    process = subprocess.Popen(bashCommand.split(), stdout=subprocess.PIPE)
    output, error = process.communicate()

    yield output

futures = [longProcess() for x in range(0, 5)]

while True:
    if all(x.done() == True for x in futures):
        break
    time.sleep(1)

print('All futures resolved')

The problem is that x.done() is returning True for all futures before my bash command is finished.

How can I turn process.communicate() into a future (Only done once the keyword "hi" is available) so that I can wait on all the futures to be done and get then get the output from the futures?

tornado python-2.7
1 Answers

Use Tornado's own process.Subprocess class which is a wrapper around stdlib's subprocess.Popen.

Example:

from tornado import process

@gen.coroutine
def longProcess():
    bashCommand = "sleep 5 && echo hi"

    proc = process.Subprocess(bashCommand.split(), stdout=subprocess.PIPE)

    yield proc.wait_for_exit() # `wait_for_exit` returns a future 
                               # which you can yield. Basically, it means
                               # you can wait for the process to complete
                               # without blocking the server

    return proc.stdout.read() # return the result of the process

    # for Python 2:
    # you can't return from a generator
    # instead use this:
    # raise gen.Return(proc.stdout.read())

You don't need the while loop. You can move that code inside some other coroutine and yield the list of futures. Like this:

@gen.coroutine
def main():
    futures = [longProcess() for x in range(0, 5)]

    results = yield futures # this coroutine will not move further 
                            # until every future in the list 
                            # `futures` is resolved

    for result in results:
        print(result)

    print('All futures resolved')

As a side note, don't use time.sleep. It will block the whole server. Instead use the asynchronous equivalent gen.sleep - yield gen.sleep(1)

user_19252570
Related