I have been scraping a website with BeatifulSoup. However, after a high number of queries (around 110 thousand) my IP address got blocked. With this in mind, I have been looking into IP rotation. I tried to implement the following code that I found here:
import requests
from bs4 import BeautifulSoup
from random import choice
def proxy_generator():
response = requests.get("https://sslproxies.org/")
soup = BeautifulSoup(response.content, 'html5lib')
proxy = {'https': choice(list(map(lambda x:x[0]+':'+x[1], list(zip(map(lambda x:x.text, soup.findAll('td')[::8]), map(lambda x:x.text, soup.findAll('td')[1::8]))))))}
return proxy
def data_scraper(request_method, url, **kwargs):
while True:
try:
proxy = proxy_generator()
print("Proxy currently being used: {}".format(proxy))
response = requests.request(request_method, url, proxies=proxy, timeout=7, **kwargs)
break
# if the request is successful, no exception is raised
except:
print("Connection error, looking for another proxy")
pass
return response
As you can see, this solution uses the free IPs provided at https://sslproxies.org/. However, I am just obtaining the following output:
Proxy currently being used: {'https': ' FR:118'}
Connection error, looking for another proxy
Proxy currently being used: {'https': '84.53.243.83:3128'}
Connection error, looking for another proxy
Proxy currently being used: {'https': ' IS:10'}
Connection error, looking for another proxy
Proxy currently being used: {'https': '139.180.194.41:80'}
Connection error, looking for another proxy
Proxy currently being used: {'https': '139.155.172.148:8889'}
Connection error, looking for another proxy
Proxy currently being used: {'https': ' LT:21'}
Connection error, looking for another proxy
Proxy currently being used: {'https': '186.152.139.187:10809'}
Connection error, looking for another proxy
Proxy currently being used: {'https': '213.230.69.33:8080'}
Connection error, looking for another proxy
Proxy currently being used: {'https': '169.57.1.85:80'}
Connection error, looking for another proxy
Ad infinitum! My guess is that this is happening because the IPs provided are used by a high quantity of users and hence become useless quite fast. Do you know any good alternatives for IP providers? Ideally, if they are free that´s better, but I have also seen some paid ones like Zyte.
PD: if you think that there is no need for IP rotation (e.g: I could just delay my code intentionally in order for it to appear human-like) let me know in the comments :)