stale element even if using WebDriverWait

Question

I do not understand why I am getting this error :

raise exception_class(message, screen, stacktrace) selenium.common.exceptions.StaleElementReferenceException: Message: The element reference of is stale; either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed

I'm using WebDriverWait twice to check if new page is loaded :

If pagination link for the new page changes accordingly
If the new page's course list div element is loaded


    from selenium import webdriver
    from selenium.webdriver.common.keys import Keys
    from selenium.webdriver.firefox.options import Options
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    from selenium.common.exceptions import TimeoutException
    
    def waitForLoad(inputXPath): 
        Wait = WebDriverWait(driver, 10)
        Wait.until(EC.presence_of_element_located((By.XPATH, inputXPath)))
    
    options = Options()
    options.headless = True
    driver = webdriver.Firefox(options=options, service_log_path='NUL')
    
    driver.get("https://www.udemy.com/courses/development/?sort=highest-rated")
    
    courses = []
    f = open("0udemy.txt","a", encoding="utf-8")
    page = 1
    
    try:
        waitForLoad("//div[@class='filter-panel--paginated-course-list--2F0x1']")
    except TimeoutException as e:
        print("timeout!")
    
    while True:

        ## I also tried that : 
        #courses = driver.find_elements_by_xpath("//div[@class='course-list--container--3zXPS']//a[contains(@class, 'browse-course-card--link--3KIkQ')]")
        #for i in courses:
        #    f.write(f"{i.get_attribute('href')}
")

        for i in range(16):
            f.write(driver.find_elements_by_xpath("//div[@class='course-list--container--3zXPS']//a[contains(@class, 'browse-course-card--link--3KIkQ')]")[i].get_attribute('href')+"
")
    
        if len(driver.find_elements_by_xpath("//a[@class='udlite-btn udlite-btn-small udlite-btn-secondary udlite-heading-sm udlite-btn-icon udlite-btn-icon-small udlite-btn-icon-round pagination--next--5NrLo']"))==0 :
            break
    
        driver.find_elements_by_xpath("//a[@class='udlite-btn udlite-btn-small udlite-btn-secondary udlite-heading-sm udlite-btn-icon udlite-btn-icon-small udlite-btn-icon-round pagination--next--5NrLo']")[0].click()
    
        page+=1
        try:
            waitForLoad(f"//a[@class='udlite-btn udlite-btn-small udlite-btn-ghost udlite-heading-sm pagination--page--3FKqV pagination--active--3BrK7' and text()={page}]")
        except TimeoutException as e:
            print("timeout!")
            break   
        try:
            waitForLoad("//div[@class='filter-panel--paginated-course-list--2F0x1']")
        except TimeoutException as e:
            print("timeout!")
            break
    
    f.close()
    driver.close()

I'm getting the stale error between page 20 and 80. My world record is page 78.

EDIT :

Thanks to arundeep, I can continue the script from where it got the stale error, using a try/except.

When I encounteer the error, I'm catching the exception, closing the driver, creating a new driver instance, get the page where I got the error, and continue the while loop from that page :

    except StaleElementReferenceException as e:
        print('stale:', e, 'page: ', page)
        driver.close()
        driver = webdriver.Firefox(options=options, service_log_path='NUL')
        driver.get(f"https://www.udemy.com/courses/development/?sort=highest-rated&p={page}")

I do get a stale error at every 10 page or so. And I'm still wondering why the error happens.

I abandon the idea. Always getting duplicated course links. If someone can explain me the stale error...

user_9901261 · Answer

I just went through pages 1 at a time and printed(you can use f.write). You need to add a time.sleep() so Selenium doesn't crash. This can go forever until it runs out of pages. Or if specify a if page==n: break. It can even go through the seleniums life cycle.

while True:
    try:
        courses=WebDriverWait(driver, 30).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='course-list--container--3zXPS']//a[contains(@class, 'browse-course-card--link--3KIkQ')]")))
        for course in courses:
            print(course.get_attribute('href')+"
")
        driver.find_elements_by_xpath("//a[@class='udlite-btn udlite-btn-small udlite-btn-secondary udlite-heading-sm udlite-btn-icon udlite-btn-icon-small udlite-btn-icon-round pagination--next--5NrLo']")[0].click()
        page=page+1
        time.sleep(5)
    except:
        url=driver.current_url
        driver.close()
        driver = webdriver.Chrome(ChromeDriverManager().install(),options=options)
        driver.get(url)

stale element even if using WebDriverWait

1 Answers