**Python selenium error** :Message: stale element reference: element is not attached to the page document

user_10077354 0
Viewed 5708

I am trying to fetch the tweets from a twitter url. But when I execute the code, the following error occurs some of the times :

StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
  (Session info: chrome=84.0.4147.89)

The error points to this line :

---> 18         print(tweet.text)

This is the corresponding code :

import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import csv

browser = webdriver.Chrome('/Users/Suraj/Desktop/twitter/chromedriver')
hashtag = 'Candles4SSR'
url = 'https://twitter.com/hashtag/'+hashtag+'?src=hashtag_click'
browser.get(url)
time.sleep(1)
body = browser.find_element_by_tag_name('body')

tweet_contents = []
for _ in range(5):
    tweets = browser.find_elements_by_css_selector("[data-testid=\"tweet\"]")
    
    for tweet in tweets:
        print(tweet.text)
        tweet_contents += [tweet.text]
    body.send_keys(Keys.PAGE_DOWN)
    time.sleep(0.4)

Any help or suggestions would be really appreciated.

web-scraping selenium python-3.x python
3 Answers

Stale element exception means Web element is there on page but driver instance could not interact with that element.

There  are 2 ways to overcome this stale element issue 
1.Using refresh method in selenium (driver.navigate().refresh();)
2.Using for loop try to looping until that element is click
user_60599670

The web element tweets found before the for loop becomes inaccessible inside the for loop at the end for some reason.

So I tried initiating tweets again inside the for loop and it works now. It seems to be a temporary fix, but works.

The last block of my code now :

tweet_contents = []
for _ in range(5):
    tweets = browser.find_elements_by_css_selector("[data-testid=\"tweet\"]")
    time.sleep(1)
    for tweet in tweets:
        print(tweet.text)
        tweet_contents += [tweet.text]
        tweets = browser.find_elements_by_css_selector("[data-testid=\"tweet\"]") # initialising tweets variable again inside for loop
    body.send_keys(Keys.PAGE_DOWN)
    time.sleep(0.4)

tweet_contents = sorted(set(tweet_contents), key=tweet_contents.index) # To remove duplicates, whilst preserving the order of tweets
user_100773540

seems like it takes the tweets to load more than 0.4 seconds

i copy pasted the code with changing the sleep time from 0.4 to 2 seconds and it worked fine.

browser = webdriver.Chrome('/Users/Suraj/Desktop/twitter/chromedriver')
hashtag = 'Candles4SSR'
url = 'https://twitter.com/hashtag/' + hashtag + '?src=hashtag_click'
browser.get(url)
time.sleep(1)
body = browser.find_element_by_tag_name('body')

tweet_contents = []
for _ in range(5):
    tweets = browser.find_elements_by_css_selector("[data-testid=\"tweet\"]")

    for tweet in tweets:
        print(tweet.text)
        tweet_contents += [tweet.text]
    body.send_keys(Keys.PAGE_DOWN)
    time.sleep(2)

if you see it works on your env consider increasing the sleep time from 0.4 to some higher number.

another option is to use a retry mechanism.

user_71039060
Related