broken pipe with aiohttp

user_3026886 0
Viewed 287

i have this method based here:

async def check_if_200(url):
    async with aiohttp.ClientSession(connector=aiohttp.TCPConnector(limit=0)) as session:
         async with session.head(url) as response:
            return response == 200

being called from 200 coroutines, in a infinite loop. sometimes, randomly, i get:

  File "/myProjectPath/myScript.py", line 71, in check_if_200
    async with session.head(url) as response:
  File "/myPyenvPath/lib/python3.9/site-packages/aiohttp/client.py", line 1138, in __aenter__
    self._resp = await self._coro
  File "/myPyenvPath/lib/python3.9/site-packages/aiohttp/client.py", line 559, in _request
    await resp.start(conn)
  File "/myPyenvPath/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 898, in start
    message, payload = await protocol.read()  # type: ignore[union-attr]
  File "/myPyenvPath/lib/python3.9/site-packages/aiohttp/streams.py", line 616, in read
    await self._waiter
aiohttp.client_exceptions.ClientOSError: [Errno 32] Broken pipe

i know what a broken pipe is, but i don't know how i could fix this

aiohttp python
1 Answers

Try to use a Semaphore to make requests, like:

async def safe_request(semaphore, url):
   async with semaphore:
      return await check_if_200(url)

async def check_if_200(url):
    async with aiohttp.ClientSession() as session:
         async with session.head(url) as response:
            return response == 200

async def main():
    semaphore = asyncio.Semaphore(10) # number of parallel requests
    tasks = list()
    tasks.append(asyncio.ensure_future(safe_request(semaphore, url))

    results = await asyncio.gather(*tasks)
user_188828060
Related