Python program runs async fetching task very fast on one machine but not on others

Viewed 87

I have an issue where python code runs super fast on my PC while I already tried another 4 machines and all of them run the same code way slower.

On the PC that I run the code it goes through 200 requests and returns 200 responses in 3 minutes. There is nothing special about this machine.

i5 gen 5 8GB RAM SSD drive

While I have already tried on the more powerful machine and less powerful one and when I run the same code it is way slower and on the slower machine it only does 10 records before it exits like it has finished. On the faster machine it goes up to 35 records and exits then.

Any idea what could be causing the issue?

My machine has more redistributable versions. Can they give an advantage?

Below is the code that is involved in fetching part.

async def fetch(session, payload):
    url = ApiUrl
    #print(payload)
    async with session.get(url, data = payload) as response:
        readyResponse= await response.text()
        global c
        c+=1
        print(c)
        parseIt(readyResponse) # function that parses results


async def fetch_all(payloads, loop):
    connector = aiohttp.TCPConnector(limit=10)
    async with aiohttp.ClientSession(connector=connector, loop=loop) as session:
        results = await asyncio.gather(*[fetch(session, payload) for payload in allReq], return_exceptions=True)
        return results


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    payloads = allReq
    htmls = loop.run_until_complete(fetch_all(payloads, loop))


1 Answers

I found an answer.

I had to install pip install aiohttp[speedups] which includes aiodns for speeding up DNS resolving by client API and aiodns and brotlipy in one bundle.

That's it.

Related