Selenium is unable to locate elements by class name

Admin 1
Viewed 184

I am trying to get a list of the prices from this page.

The class name of the elements I'm trying to get is called s-item__price. This is my code:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

url = 'https://www.ebay.de/sch/i.html?_from=R40&_nkw=iphone+8+&_sacat=0&LH_TitleDesc=0&LH_ItemCondition=3000&rt=nc&LH_Sold=1&LH_Complete=1'

chrome_options = Options()
chrome_options.add_argument('--headless')

browser = webdriver.Chrome(options=chrome_options)

browser.get(url)

print(browser.find_elements_by_class_name('s-item__price'))

browser.quit()

The output is just an empty list.

selenium-webdriver web-scraping selenium python
2 Answers

You can use WebDriverWait to wait until the javascript generated the element:

wait = WebDriverWait(browser, 15) # 15 sec timeout
wait.until(expected_conditions.visibility_of_element_located((By.CLASS_NAME, 's-item__price')))

You could also use presence_of_elements_located but if it comes to click interaction it won't work with hidden elements. So prefer using: visibility_of_element_located

Example Code:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions

url = 'https://www.ebay.de/sch/i.html?_from=R40&_nkw=iphone+8+&_sacat=0&LH_TitleDesc=0&LH_ItemCondition=3000&rt=nc&LH_Sold=1&LH_Complete=1'

options = Options()
options.add_argument('--headless')

browser = webdriver.Chrome(options=options)
browser.get(url)

wait = WebDriverWait(browser, 15) # Throws a TimeoutException after 15 seconds
wait.until(expected_conditions.visibility_of_element_located((By.CLASS_NAME, 's-item__price')))
# you may also culd use the return value of the wait

print(browser.find_elements_by_class_name('s-item__price'))
browser.quit()
user_135046580

You get an empty list I think it because you need wait.

Use WebDriverWait and utilize .presence_of_all_elements_located to collect elements in a list.

Then extract them with a loop and you must call the .text method to grab the text

browser.get('https://www.ebay.de/sch/i.html?_from=R40&_nkw=iphone%208%20&_sacat=0&LH_TitleDesc=0&LH_ItemCondition=3000&rt=nc&LH_Sold=1&LH_Complete=1')
wait = WebDriverWait(browser, 20)
list_price = wait.until(EC.presence_of_all_elements_located((By.CLASS_NAME, 's-item__price')))
for price in list_price:
    print(price.text)
driver.quit()

Following import:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
user_87520620
Related