aiohttp-socks create_connection to switch TOR IPs for each request

Viewed 132

I am trying to download 20 thousand pages from one url and I need for my IP to switch every two thousand(or so) requests to not get ssl errors or get "knocked out"

I have presently resorted to asking two thousand pages at a time which is sub-optimal as you can imagine.

My code reads:

      for cnt, ticker in enumerate(sif_df["ticker"][begloc_int:endloc_int]): 
         composite_urlstr = urlstr + ticker + suffix_urlstr
          #print(composite_urlstr)
         html_list.append(fetch(session, composite_urlstr))
      soups = await asyncio.gather(*html_list)

so I am now manually changing begloc_int and endloc_int

I was thinking of adding something like:

if cnt % 200 == 0:
   switchIP() 

But the version of swithcIP() that I have works for a regular Tor request not a asynced one.

I was told that what I can use is aiohttp-socks create_connection but I don't know how to do that and looking on the github entry does not clarify much as there is no example. I think the create_connection needs to happen within the fetch function, but I simply don't know.

async def fetch(session,url):  
    # Somewhere in here stick in a create_connection

    # with async_timeout.timeout(10):
    async with session.get(url) as response:        
        return BeautifulSoup(await response.text(),"html.parser")

If anybody knows how to do this I would greatly appreciate the help.

Thank you very much in advance

0 Answers
Related