How to create a queue for python-requests in Django?

Viewed 469

REST API service has a limit of requests (say a maximum of 100 requests per minute). In Django, I am trying to allow USERs to access such API and retrieve data in real-time to update SQL tables. Therefore there is a problem that if multiple users are trying to access the API, the limit of requests is likely to be exceeded.

Here is a code snippet as an example of how I currently perform requests - each user will add a list of objects he wants to request and run request_engine().start(object_list) to access the API. I use multithreading to speed up requests. I also allow retrying failed API requests via setting a limit on the number of requests for each request object upper_limit.

As I understand there should be some queue for API requests. I anticipate there must be a more elegant solution for this, however, I could not find any similar examples. How can one implement/rewrite this for multiUSER usage with Django?

import requests
from multiprocessing.dummy import Pool as ThreadPool

N=50 # number of threads
upper_limit=1 # limit on the number of requests for a single object

class request_engine():
    def __init__(self):
        pass

    def start(self,objs):
        self.objs={obj:{'status':0,'data':None} for obj in objs}
        done=False
        while not done:
            self.parallel_requests()
            done=all(_['status']>upper_limit or _['status']==-1 for obj,_ in self.objs.items())
        return dict(self.objs)

    def single_request(self,request_obj):
        URL = f"https://reqres.in/api/users?page={request_obj}"
        r = requests.get(url = URL)
        if r.ok:
            res = r.json()
            self.objs[request_obj]['status']=-1
            self.objs[request_obj]['data']=res
        else:
            self.objs[request_obj]['status']+=1

    def parallel_requests(self):
        objs=[obj for obj,_ in self.objs.items() if _['status']!=-1 and _['status']<=upper_limit]

        pool = ThreadPool(N)
        pool.map(self.single_request, objs)
        pool.close()
        pool.join()


objs=[1,2,3,4,5,6,7,7,8,234,124,24,535,6,234,24,4,1,3,4,5,4,3,5,3,1,5,2,3,5,3]
result=request_engine().start(objs)
print([_['status'] for obj,_ in result.items()])
# status corresponds to the number of unsuccessful requests
# status=-1 implies success of the request

Thanks in advance.

0 Answers
Related