I am scraping a website using a combination of python, selenium and TOR.
Scraping works fine, but it becomes significantly slower over time, initially scraping up to 30 pages a minute, and then, slowly but steadily, declining to around 4-5 pages a minute within a time frame of a couple of minutes.
I am not currently concerned with parsing the contents of the web pages. I will do this at a later stage, so this is beyond the scope of this question.
Here is my code:
import requests
from stem import Signal
from stem.control import Controller
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
def get_tor_session():
session = requests.Session()
# Tor uses the 9050 port as the default socks port
session.proxies = {'http': 'socks5://127.0.0.1:9050',
'https': 'socks5://127.0.0.1:9050'}
#session.headers = {'User-Agent': 'Mozilla/5.0'}
session.headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux i686; G518Rco3Yp0uLV40Lcc9hAzC1BOROTJADjicLjOmlr4=) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36','Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8','Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3','Accept-Encoding': 'gzip, deflate, sdch','Accept-Language': 'en-US,en;q=0.8','Connection': 'keep-alive'}
return session
# signal TOR for a new connection
def renew_connection():
with Controller.from_port(port = 9051) as controller:
controller.authenticate(password="16:8F6BA4CFC409D432607A853D139F3FFE43A1B77E4893BEA3CE7F63DA27")
controller.signal(Signal.NEWNYM)
chrome_options = Options()
chrome_options.add_argument("--proxy-server=socks5://127.0.0.1:9050")
url = "https://www.targesite.com/"
driver = webdriver.Chrome(options=chrome_options)
for i in range(1,1000000,1):
renew_connection()
session = get_tor_session()
finalUrl = url + str(i) + "/"
driver.get(finalUrl)
f=open("C:\\outputs\\" + str(i) +".txt", "a+", encoding="utf-8")
f.write(driver.page_source)
f.close()
I don't have too much experience with scraping in python, and am fairly rusty in my python in general - therefore I am unsure what is causing these slowdowns. My question is:
What can I do to avoid these slowdowns? With "correct" code, I should be able to maintain the high initial scraping rate of 30 pages a minute, or not?