Web element reference not seen before

user_12266614 0
Viewed 2581

I am trying to make a web-scraper that fetches all the results of every google search, however it kept outputting a "web element reference not seen before error". I assume it's due to the code trying to find the element before the url loads, but i am not too sure how to fix it.

from selenium import webdriver

#number of pages
max_page = 5

#number of digits (ie: 2 is 1 digit, 10 is 2 digits)
max_dig = 1

#Open up firefox browser
driver = webdriver.Firefox()

#inputs search into google 
question = input("\n What would you like to google today, but replace every space with a '+' (ie: search+this)\n\n")

search = [] 

#get multiple pages
for i in range(0, max_page + 1):
    #inserts page number into google search
    page_num = (max_dig - len(str(i))) * "0" + str(i)
    #inserts search input and cycles through pages
    url =  "https://www.google.com/search?q="+ question +"&ei=LV-uXYrpNoj0rAGC8KSYCg&start="+ page_num +"0&sa=N&ved=0ahUKEwjKs8ie367lAhUIOisKHQI4CaM4ChDy0wMIiQE&biw=1356&bih=946"
    #finds element in every search page
    search+=(driver.find_elements_by_class_name('LC20lb'))
    driver.get(url)

#print results
search_items = len(search)
for a in range(search_items):
    #print the page number
    print(type(search[a].text))

Traceback (most recent call last):
  File "screwdriver.py", line 32, in <module>
    print(type(search[b].text))
selenium.common.exceptions.NoSuchElementException: Message: Web element reference not seen before: 6187cf00-39c8-c14b-a2de-b1d24e965b65
web-scraping selenium python
1 Answers

Problem is that Selenium doesn't keep HTML which you found but rather referece to element on current page. When you load new page - get() - then reference try to find element on new page and it can't find it. You should get text (and any other information) from item before you load new page.

from selenium import webdriver

max_page = 5

driver = webdriver.Firefox()

question = input("\n What would you like to google today, but replace every space with a '+' (ie: search+this)\n\n")

search = [] 

for i in range(max_page+1):
    page_num = str(i)

    url =  "https://www.google.com/search?q="+ question +"&ei=LV-uXYrpNoj0rAGC8KSYCg&start="+ page_num +"0&sa=N&ved=0ahUKEwjKs8ie367lAhUIOisKHQI4CaM4ChDy0wMIiQE&biw=1356&bih=946"

    items = driver.find_elements_by_class_name('LC20lb')

    for item in items:
        search.append(item.text) 

    driver.get(url)

for item in search:
    print(item)
user_18320580
Related