Stalemate Exception Selenium Python while switching Url

user_14559344 0
Viewed 39

I'm Getting an error while implementing the following scenario: Driver goes to a url gets a list of elements and iterates over them, for each iteration it has to go on another link and get some information, This is working fine but once we come back from the page in the next iterations all the previous elements stored seems to be throwing the following exception: "StaleElementReferenceException: Message: stale element reference: element is not attached to the page document"

Following is my code:

driver.get('https://www.jobs.nhs.uk/xi/search_vacancy/')

input=driver.find_elements(By.XPATH,'//*[@id="typeahead_key"]')
if (len (input)>0):
    input[0].send_keys('nurse')
else:
    print ('Not found')

button = driver.find_element(By.NAME, 'searchBtn1')
button.click()

i=1
d=[]
while (i<=6):
    vacancies= driver.find_elements(By.XPATH,"//*[@class='vacancy']")
    for vac in vacancies:
        print (len(vacancies))
        print (vacancies[1].text)
        print("-----------------------"+str(i)+"-------------------------")
        ls = vac.text.split("\n")
        print (ls[10][14:])
        driver.get('https://beta.jobs.nhs.uk/candidate/jobadvert/'+ls[10][14:])
        content=driver.find_element(By.XPATH,"//*[@class='nhsuk-grid-column-one-third']")
        print (content.text)
        # print (vac.text)
        # d.append(vac.text)
        driver.back()


        print("----------------------------------------------------")
    i+=1
    try:
        driver.find_element(By.XPATH,'//*[@title="Next page"]').click()
    except WebDriverException:
        exit(1)

Error Log:

Traceback (most recent call last):
  File "C:\Users\Lenovo T14\Documents\jo\sel\sample.py", line 46, in <module>
    print (vacancies[1].text)
  File "C:\Users\Lenovo T14\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\selenium\webdriver\remote\webelement.py", line 85, in text
    return self._execute(Command.GET_ELEMENT_TEXT)['value']
  File "C:\Users\Lenovo T14\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\selenium\webdriver\remote\webelement.py", line 773, in _execute
    return self._parent.execute(command, params)
  File "C:\Users\Lenovo T14\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\selenium\webdriver\remote\webdriver.py", line 430, in execute
    self.error_handler.check_response(response)
  File "C:\Users\Lenovo T14\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\selenium\webdriver\remote\errorhandler.py", line 247, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
  (Session info: chrome=103.0.5060.114)
Stacktrace:
Backtrace:
        Ordinal0 [0x00FDD953+2414931]
        Ordinal0 [0x00F6F5E1+1963489]
        Ordinal0 [0x00E5C6B8+837304]
        Ordinal0 [0x00E5F0B4+848052]
        Ordinal0 [0x00E5EF72+847730]
        Ordinal0 [0x00E5F200+848384]
        Ordinal0 [0x00E84FDF+1003487]
        Ordinal0 [0x00EA449C+1131676]
        Ordinal0 [0x00E7FC74+982132]
        Ordinal0 [0x00EA46B4+1132212]
        Ordinal0 [0x00EB4812+1198098]
        Ordinal0 [0x00EA42B6+1131190]
        Ordinal0 [0x00E7E860+976992]
        Ordinal0 [0x00E7F756+980822]
        GetHandleVerifier [0x0124CC62+2510274]
        GetHandleVerifier [0x0123F760+2455744]
        GetHandleVerifier [0x0106EABA+551962]
        GetHandleVerifier [0x0106D916+547446]
        Ordinal0 [0x00F75F3B+1990459]
        Ordinal0 [0x00F7A898+2009240]
        Ordinal0 [0x00F7A985+2009477]
        Ordinal0 [0x00F83AD1+2046673]
        BaseThreadInitThunk [0x76136739+25]
        RtlGetFullPathName_UEx [0x77C18FEF+1215]
        RtlGetFullPathName_UEx [0x77C18FBD+1165]

Update: I have found a workaround by opening further links into other tabs, Get the information needed and switch back to the previous tab.

selenium-webdriver web-scraping selenium exception python
2 Answers

DOM Issue:

This has to do with the vacancies element not being attached to the DOM. See Selenium Docs on this error.

A common technique used for simulating a tabbed UI in a web app is to prepare DIVs for each tab, but only attach one at a time, storing the rest in variables. In this case, it's entirely possible that your code might have a reference to an element that is no longer attached to the DOM (that is, that has an ancestor which is document.documentElement).

If WebDriver throws a stale element exception in this case, even though the element still exists, the reference is lost. You should discard the current reference you hold and replace it, possibly by locating the element again once it is attached to the DOM.

You cannot perform operations on an element after the webpage has been changed. You need to re-establish your variables after every DOM change to have the element be a part of the current DOM structure so that you can interact with the elements. For your particular example, you reference "vacancies" once, yet change the webpage several times. For this I could recommend putting the vacancies variable within the loop and find by a single element:

driver.find_element(By.XPATH, ...)
user_172546770

Your workaround is the solution I would recommend. Elements on the webpage are tied to that particulate instance of the page, due to stuff involving the DOM as @LukeHamilton said. If you ever refresh a web page or navigate away and back to it you will need to relocate all elements.

I recommend using your workaround. Just open a new window or tab and switch between them.

main_tab = driver.curren_window_handle
while (i<=6):
    vacancies= driver.find_elements(By.XPATH,"//*[@class='vacancy']")
    for vac in vacancies:
        print (len(vacancies))
        print (vacancies[1].text)
        print("-----------------------"+str(i)+"-------------------------")
        ls = vac.text.split("\n")
        print (ls[10][14:])
        driver.switch_to.new_window('tab')
        driver.get('https://beta.jobs.nhs.uk/candidate/jobadvert/'+ls[10][14:])
        content=driver.find_element(By.XPATH,"//*[@class='nhsuk-grid-column-one-third']")
        print (content.text)
        # print (vac.text)
        # d.append(vac.text)
        driver.close()
        driver.switch_to.window(main_tab)
user_123343530
Related