Python Selenium Stale Element Hangs Forever

Question

I'm using Python Selenium to extract all links from a page and I chose Selenium because many of the sites I'm targeting use JS to render the DOM. However, Selenium is throwing a Stale Exception error when traversing the page elements.

import from selenium.webdriver import Firefox

def request(url):
     urls = []
     browser = Firefox()
     browser.implicitly_wait(7)
     browser.get(url)

     elements = browser.find_elements_by_xpath("//a")
     for link in elements:
            href = link.get_attribute('href')
            urls.append(href)

     browser.close()
     return urls

This is the error selenium throws

File "/var/extractor/main.py", line 80, in request
    href = link.get_attribute('href')
File "/usr/local/lib/python3.4/dist-packages/selenium/webdriver/remote/webelement.py", line 113, in get_attribute
    resp = self._execute(Command.GET_ELEMENT_ATTRIBUTE, {'name': name})
File "/usr/local/lib/python3.4/dist-packages/selenium/webdriver/remote/webelement.py", line 469, in _execute
    return self._parent.execute(command, params)
File "/usr/local/lib/python3.4/dist-packages/selenium/webdriver/remote/webdriver.py", line 201, in execute
    self.error_handler.check_response(response)
File "/usr/local/lib/python3.4/dist-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: Element not found in the cache - perhaps the page has changed since it was looked up

So to try to fix this I wrapped the get_attribute('href') in a try exception.

     for link in elements:
            try:
                href = link.get_attribute('href')
            except StaleElementReferenceException:
                continue
            urls.append(href)

But that didn't work either! The application never exits the for loop, it just hangs forever. So I'm not quite sure what to do. I've seen a lot of other posts regarding Stale Element Exception but none that hang forever. I feel like this whole StaleElement exception is a design flaw in selenium and it should be able to fail more gracefully then this. Would appreciate any suggestions on how to fix this.

[EDIT]: Until I figure this out, I'm sending the page_source to lxml to parse links instead of Selenium. This does solve this issue for now. But I'd prefer to not introduce another dependency and fix the issue with Selenium.

Python Selenium Stale Element Hangs Forever

0 Answers