stale element reference: element is not attached to the page document in selenium python

ScrapperMaster 0
Viewed 34

I am looping through products and extracting some information and in loop i am also clicking on the product to go to Product Detail page (PDP). After that i am extracting brand information and want to go back to the previous page. but after using the driver.goback() the loop does not work.from selenium import webdriver

from selenium.common import exceptions
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
import pandas as pd
import time
PATH = r"C:\Users\MaxPain\Documents\geckodriver.exe"

ser = Service(PATH_C)

driver = webdriver.Chrome(service=ser)

# driver.execute_script("window.scrollBy(0, 2000)")
MainURL = "https://www.amazon.com/s?i=electronics-intl-ship&bbn=16225009011&rh=n%3A2811119011%2Cn%3A2407755011&dc&ds=v1%3AC0LTQbPEHkj0izU%2BGQ%2FVcuqm26QN2oitQYzfpo09qvk&qid=1663587128&rnid=2811119011&ref=sr_nr_n_2"
driver.get(MainURL)

mainloop = driver.find_elements(By.XPATH , '//div[@data-component-type="s-search-result"]')
print("Total Products : " , len(mainloop))
data = []
for index, l in enumerate(mainloop):

    title = l.find_element(By.XPATH ,'.//h2/a/span').text
    URL = l.find_element(By.XPATH, './/h2/a').get_attribute('href')
    l.find_element(By.XPATH, './/h2/a').click()
    brand = driver.find_element(By.XPATH, '//a[@id = "bylineInfo"]').text.replace("Visit the ","")
    # brand = brand.replace("Store","").trim()
    # print(brand)
    data.append({'TItle':title,"ProductURL":URL, "Brand":brand})
    driver.back()
df = pd.DataFrame(data)
df.to_csv('AmzTest.csv')

But using this code i am getting this error

selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
  (Session info: chrome=105.0.5195.127)
Stacktrace:
Backtrace:
    Ordinal0 [0x00CDDF13+2219795]
    Ordinal0 [0x00C72841+1779777]
    Ordinal0 [0x00B8423D+803389]
    Ordinal0 [0x00B86D04+814340]
    Ordinal0 [0x00B86BC2+814018]
    Ordinal0 [0x00B86E50+814672]
    Ordinal0 [0x00BB2D4F+994639]
    Ordinal0 [0x00BB31EB+995819]
    Ordinal0 [0x00BA9531+955697]
    Ordinal0 [0x00BCE844+1108036]
    Ordinal0 [0x00BA94B4+955572]
    Ordinal0 [0x00BCEA14+1108500]
    Ordinal0 [0x00BDF192+1175954]
    Ordinal0 [0x00BCE616+1107478]
    Ordinal0 [0x00BA7F89+950153]
    Ordinal0 [0x00BA8F56+954198]
    GetHandleVerifier [0x00FD2CB2+3040210]
    GetHandleVerifier [0x00FC2BB4+2974420]
    GetHandleVerifier [0x00D76A0A+565546]
    GetHandleVerifier [0x00D75680+560544]
    Ordinal0 [0x00C79A5C+1808988]
    Ordinal0 [0x00C7E3A8+1827752]
    Ordinal0 [0x00C7E495+1827989]
    Ordinal0 [0x00C880A4+1867940]
    BaseThreadInitThunk [0x76DD7BA9+25]
    RtlInitializeExceptionChain [0x777CBB3B+107]
    RtlClearBits [0x777CBABF+191]

Is there any way to resolve this problem.

I know i can extract all the urls once and create another robot to extract PDP information. But the purpose of this is more to learn about how i can navigate over different pages.

navigation selenium-webdriver web-scraping selenium python
1 Answers

When you going to another page with this line l.find_element(By.XPATH, './/h2/a').click() all the web elements initially collected in mainloop list are becoming stale. In Selenium Web Element is actually a reference to a physical web element. When you coming back to the main page it is re-rendered so the previously collected references to web elements no more pointing to those elements.
To make your code working you need to collect the mainloop list again when getting back to the main page.
The following code works:

url = 'https://www.amazon.com/s?i=electronics-intl-ship&bbn=16225009011&rh=n%3A2811119011%2Cn%3A2407755011&dc&ds=v1%3AC0LTQbPEHkj0izU%2BGQ%2FVcuqm26QN2oitQYzfpo09qvk&qid=1663587128&rnid=2811119011&ref=sr_nr_n_2'
driver.get(url)
wait = WebDriverWait(driver, 20)

wait.until(EC.visibility_of_all_elements_located((By.XPATH, '//div[@data-component-type="s-search-result"]')))
mainloop = driver.find_elements(By.XPATH, '//div[@data-component-type="s-search-result"]')
print("Total Products : " , len(mainloop))
data = []
for index, l in enumerate(mainloop):
    wait.until(EC.visibility_of_all_elements_located((By.XPATH, '//div[@data-component-type="s-search-result"]')))
    mainloop = driver.find_elements(By.XPATH, '//div[@data-component-type="s-search-result"]')
    l = mainloop[index]
    title = l.find_element(By.XPATH, './/h2/a/span').text
    URL = l.find_element(By.XPATH, './/h2/a').get_attribute('href')
    l.find_element(By.XPATH, './/h2/a').click()
    brand = driver.find_element(By.XPATH, '//a[@id = "bylineInfo"]').text.replace("Visit the ","")
    data.append({'TItle':title,"ProductURL":URL, "Brand":brand})
    driver.back()

df = pd.DataFrame(data)
df.to_csv('AmzTest.csv')
Prophet0
Related