How to make multiple REST calls asynchronous in python3

Question

I have the following code to make multiple REST calls. Basically I have a dictionary where key is a string and value is a json date that I need to use as payload to pass to a REST API POST method.

At the moment, the dictionary contains 10 entries, so I need to make 10 REST calls.

At the moment, I have implemented using requests package in python3 which is synchronous in nature. So after 1 REST call, it waits for its response and similarly for 10 REST calls, it will wait 10 times for the response from API.

def createCategories(BACKEND_URL, token, category): url = os.path.join(BACKEND_URL, 'api/v1/category-creation')

category_dict = read_payloads(category)

headers = {
    "token": f'{token}',
    "Content-Type": "application/json",
    "accept": "application/json"
}

for name, category_payload in category_dict.items():
    json_payload = json.dumps(category_payload)
    response = requests.request("POST", url, headers=headers, data=json_payload)
    ##########################
    ## Load as string and parsing
    response_data = json.loads(response.text)
    print(response_data)

    category_id = response_data['id']
    message = 'The entity with id: ' + str(category_id) + ' is created successfully. '
    logging.info(message)

return "categories created successfully."

I read that we need to use asyncio to make these asynchronous. Please advise on the code changes.

Artiom Kozyrev · Answer

You can continue using requests library. You need to use threading or concurrent.futures modules to make several requests simutaneoudly.

Another option is to use some async library like aiohttp or some others.

import requests

from threading import current_thread
from concurrent.futures import ThreadPoolExecutor, Future
from time import sleep, monotonic

URL = "https://api.github.com/events"


def make_request(url: str) -> int:
    r = requests.get(url)
    sleep(2.0)  # wait n seconds
    return r.status_code


def done_callback(fut: Future):
    if fut.exception():
        res = fut.exception()
        print(f"{current_thread().name}. Error: {res}")
    elif fut.cancelled():
        print(f"Task was canceled")
    else:
        print(f"{current_thread().name}. Result: {fut.result()}")


if __name__ == '__main__':
    urls = [URL for i in range(20)]  # 20 tasks

    start = monotonic()

    with ThreadPoolExecutor(5) as pool:
        for i in urls:
            future_obj = pool.submit(make_request, i)
            future_obj.add_done_callback(done_callback)

    print(f"Time passed: {monotonic() - start}")

How to make multiple REST calls asynchronous in python3

1 Answers