Continue to get a 'asyncio.exceptions.TimeoutError'

user_7793916 0
Viewed 149

The example below is only with 1 element being added to the event loop, with the variable asins.

I am getting a asyncio.exceptions.TimeoutError error when the asins parameter below is 180 elements or longer.

If I create a list with any of these 180 elements, I get a successful response, which tells me the issue below is not related to the API.

Can anyone tell me how to fix this?? thank you!

import asyncio
import aiohttp
import sys
import pandas as pd

def create_params(asins_set):
    params = []
    for asin in asins_set:
        param = {
            'api_key': '...',
            'type': 'product',
            'amazon_domain': 'amazon.com',
            'asin': asin,
        }
        params.append(param)
    return params

if sys.version_info[0] == 3 and sys.version_info[1] >= 8 and sys.platform.startswith('win'):
    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

# creates a list of tasks to add to the event loop at once
def get_tasks(session, params):
    tasks = []
    for param in params:
        tasks.append(session.get(
            'https://api.rainforestapi.com/request',
            params = param
        ))
    return tasks

results = []
async def get_suggested(params):
    async with aiohttp.ClientSession() as session:
        tasks = get_tasks(session, params)
        responses = await asyncio.gather(*tasks)
        for response in responses:
            results.append(await response.json())
        return results

def get_asin_titles(asins_set):
    params = create_params(asins_set)
    r = asyncio.run(get_suggested(params))
    asins_and_titles = dict()
    for result in r:
        if result['request_info']['success'] == True:
            asin = result['request_parameters']['asin']
            title = result['product']['title']
            asins_and_titles[asin] = title
    return asins_and_titles

asins = ['b07wp7q5bf']

final = get_asin_titles(asins)
print(final)
aiohttp python-asyncio python
1 Answers

The problem could be in the high amount of simultaneous http connections. Here you try to run all 180 http requests at once

responses = await asyncio.gather(*tasks)

So, you need to limit the number of simultaneous running http requests. You can do it with asyncio.Semaphore

A semaphore manages an internal counter which is decremented by each acquire() call and incremented by each release() call. The counter can never go below zero; when acquire() finds that it is zero, it blocks, waiting until some task calls release().

So, your code will look like this

...
sem = asyncio.Semaphore(10)

async def perform_request(session, param)
    async with sem:
        return await session.get(
            'https://api.rainforestapi.com/request',
            params = param
        )

def get_tasks(session, params):
    tasks = []
    for param in params:
        tasks.append(perform_request(session, param))
    return tasks

...

Now, if you run responses = await asyncio.gather(*tasks), only 10 concurrent http requests will be run at once.

You can play with number 10 to adjust your solution to API rates. If you use sem = asyncio.Semaphore(1) all requests will be performed sequentially one-by-one.

Maybe, it will be still too fast for this API. Then, you can add sleeps to reduce your RPS even more:

async def perform_request(session, param)
    async with sem:
        await asyncio.sleep(1)
        return await session.get(
            'https://api.rainforestapi.com/request',
            params = param
        )
user_81531470
Related