Using Python threads to make thousands of calls to a slow API with a rate limit

Viewed 9960

I want to make thousands of calls to an API which is kind of slow -- tens of seconds to get a response. The only limit is that I can make at most one request per second. What's the best way to do this? I think the following code works, but I feel I should be able to make better use of the threading library somehow. I'm using python 3.3

last_job = datetime.now()
for work in work_list:
    while (datetime.now()-last_job).total_seconds() < 1 or threading.active_count() >= max_threads:
        time.sleep(.1)
    threading.Thread(target=work_function, args=[work]).start()
    last_job = datetime.now()
1 Answers
Related