I need to scrape the homepage of about 10000 different websites, looking for contact information. I already built the script, using selenium, and it works pretty well. However, I'm getting a strange behavior with the webdriver... it just gets stuck, at random times, with no errors whatsoever, in this line:
driver_pcr.get("https://" + url)
I'm not sure why. At first I thought it might be some issue with the website it's trying to open, but if I kill the execution, and restart on the same website, it gets loaded and scraped without issues. Any ideas on what could be causing this, and/or how to fix it?
Here's the driver definition I'm using:
chrome_options = webdriver.ChromeOptions()
prefs = {"profile.default_content_setting_values.notifications": 2}
chrome_options.add_experimental_option("prefs", prefs)
driver_pcr = webdriver.Chrome(chrome_options=chrome_options)
driver_pcr.set_page_load_timeout(15)