I was happily extracting information from a page, which requires clicking on a link to see the email. Everything was working fine until I turned off my laptop for a few days, I'm trying to do this task again but for some reason, it isn't working anymore :( (I never changed my code or anything). Please HELP!
Before anything, I tried to replace that click with Actions, tried to use Java, changed the webdriver, changed the waiting time, tried with CSS_SELECTOR, XPATH, etc (previously worked only with LINK_TEXT) but nothing I do seems to work. Here's my code, I censored my information:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
DRIVER_PATH = '/Users/andre/Desktop/geckodriver'
driver = webdriver.Firefox(executable_path=DRIVER_PATH)
driver.get('https://membership.icsc.com/login')
USERNAME = 'XXXXXXXXXXXX'
PASSWORD = 'XXXXXX'
#login first
login = driver.find_element(By.NAME, '_username').send_keys(USERNAME)
password = driver.find_element(By.NAME, '_password').send_keys(PASSWORD)
submit = driver.find_element(By.CSS_SELECTOR, "body > div > div > div > div > div > div > form > div.text-center > p:nth-child(1) > button").click()
main = driver.get("THEURLISTOOLONGTOPUTITHERE")
#thescraper666
def get_scarping(link):
driver.get(link)
try:
email = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.LINK_TEXT, "Click here to display email")))
email.click()
driver.implicitly_wait(5)
email2 = driver.find_element(By.XPATH, '/html/body/div[5]/div/div/div/div/div/div[2]/div[2]/div/div/div[1]/div[2]/a').text
print(email2)
except NoSuchElementException:
print("NO EMAIL")
return driver.current_url
links = ['A LIST WITH A BUNCH OF URLS']
scrapings = []
for link in links:
scrapings.append(get_scarping(link))