I am trying to download 20 thousand pages from one url and I need for my IP to switch every two thousand(or so) requests to not get ssl errors or get "knocked out"
I have presently resorted to asking two thousand pages at a time which is sub-optimal as you can imagine.
My code reads:
for cnt, ticker in enumerate(sif_df["ticker"][begloc_int:endloc_int]):
composite_urlstr = urlstr + ticker + suffix_urlstr
#print(composite_urlstr)
html_list.append(fetch(session, composite_urlstr))
soups = await asyncio.gather(*html_list)
so I am now manually changing begloc_int and endloc_int
I was thinking of adding something like:
if cnt % 200 == 0:
switchIP()
But the version of swithcIP() that I have works for a regular Tor request not a asynced one.
I was told that what I can use is aiohttp-socks create_connection but I don't know how to do that and looking on the github entry does not clarify much as there is no example. I think the create_connection needs to happen within the fetch function, but I simply don't know.
async def fetch(session,url):
# Somewhere in here stick in a create_connection
# with async_timeout.timeout(10):
async with session.get(url) as response:
return BeautifulSoup(await response.text(),"html.parser")
If anybody knows how to do this I would greatly appreciate the help.
Thank you very much in advance