I try to get some information from Quora using a python web crawler program written by aiohttp. I used aiohttp.ClientSession to build a connect session and set the timeout aiohttp.ClientTimeout(total=5). Most of the time, my program was fine, but sometimes it would be stuck e.g. I tried to use aiohttp.ClientSession.post to fetch some data from https://www.quora.com/graphql/gql_para_POST?q=EditCredentialsModalQuery and check the response status. It stuck at this step. The aiohttp.ClientTimeout(total=5) seems invalid. Here is my code and bug picture:
async with semaphore:
while req_time <= 3:
try:
async with session.post(url='https://www.quora.com/graphql/gql_para_POST?q'
'=EditCredentialsModalQuery',
headers=headers, data=payload, proxy=self.proxy) as resp:
if resp.status != 200:
print(f'{user_name}[ModalQuery]: resp.status is not 200, retrying {req_time}\n')
await asyncio.sleep(2)
req_time += 1
continue
except Exception as e:
print(f'{user_name}[ModalQuery]: Async func get_cred_from_EditCredentialsModalQuery Error: {e}, '
f'retrying {req_time}')
await asyncio.sleep(2)
req_time += 1
continue
