Asynchronous http requests takes the same time as the synchronous one

user_18278142 0
Viewed 133

i'm trying to scrape a website using aiohttp and asyncio in python, but it takes the same amount of time doing it in the synchronous way. This is the asynchronous code that i'm using:

base_url="http://example.com/"
lista_link=['Scheda.aspx?codprod=122017593', 'Scheda.aspx?codprod=122017591', 'Scheda.aspx?codprod=122017590', 'Scheda.aspx?codprod=122017589', 'Scheda.aspx?codprod=122017588', 'Scheda.aspx?codprod=122017587', 'Scheda.aspx?codprod=122017586', 'Scheda.aspx?codprod=122015316']

def get_tasks(sessione):
    tasks = []
    for x in range(0,len(lista_link)):
        tasks.append(sessione.get(url=f"{base_url}{lista_link[x]}"))
    return tasks
    

async def get_sites():
    async with aiohttp.ClientSession(cookies=session_cookies) as sessione:
        tasks = get_tasks(sessione=sessione)
        risposte = await asyncio.gather(*tasks)
        for risposta in risposte:
            pagina = await risposta.text()
            lista_pagine.append(pagina)
    
 asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
 asyncio.run(get_sites())

time: 90.8

this is the synchronous one

for x in range(0,len(lista_link)):
    print(f"connessione {x} di {n_prodotti}") 
    pagina_prodotto = requests.get(f"{base_url}{lista_link[x]}",cookies=session_cookies).content
    lista_pagine.append(pagina_prodotto)

time: 88.3

Am I doing something wrong? I am passing the session-cookies from a connection that i made before for authentication, can it influence the rest of the script in any way ?

python-requests python-asyncio web-scraping asynchronous python-3.x
1 Answers

Your example uses asyncio.gather() for concurrent HTTP connections establishing but after that iterates to get HTTP response bodies sequentially.

The possible solution is:

base_url="http://example.com/"
lista_link=['Scheda.aspx?codprod=122017593', 'Scheda.aspx?codprod=122017591', 'Scheda.aspx?codprod=122017590', 'Scheda.aspx?codprod=122017589', 'Scheda.aspx?codprod=122017588', 'Scheda.aspx?codprod=122017587', 'Scheda.aspx?codprod=122017586', 'Scheda.aspx?codprod=122015316']

async def get_body(sessione, url):
    async with sessione.get(url) as resp:
        return await resp.text()
  
async def get_sites():
    async with aiohttp.ClientSession(cookies=session_cookies) as sessione:
        tasks = [get_resp(sessione, f"{base_url}{link}") for link in lista_link]
        risposte = await asyncio.gather(*tasks)
        for pagina in risposte:
            lista_pagine.append(pagina)
    
 asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
 asyncio.run(get_sites())
user_34548790
Related