Python Web Scraping an element- try until it works, then wait, and try again

user_16600299 0
Viewed 198

The code below runs perfectly, but after 10 seconds I get an exception (which says the element is not found), and thats because the element is getting refreshed. it takes less than 1 seconds to get refreshed. So all I need to do, is wait when its getting refreshed, and then continue.

items = driver.find_elements_by_class_name('box')

for item in items:
    while True:
        try:
            title = item.find_element_by_class_name('title').text
            break
        except:
            time.sleep(4)
            pass

I tried to solve it with try: except: but once it joins the except part, it gets stuck in the loop. How do we solve this?

selenium-webdriver try-except web-scraping selenium python
1 Answers

I would probably have a if condition with find_elements like below :

for item in items:
  while True:
    try:
      if len(driver.find_elements(By.CLASS_NAME, "title")) > 0:
        title = item.find_element_by_class_name('title').text
        break
      else:
        print("Elemennt could not be found")
    except NoSuchElementException as error:
      time.sleep(4)
      print("Something went wrong")
      pass

You may need this import :

from selenium.common.exceptions import TimeoutException, WebDriverException, NoSuchElementException
user_94440730
Related