web scraping selenium find elements by class name

user_11455772 0
Viewed 611

I try to extract the all the products data from this page:

https://www.shufersal.co.il/online/he/קטגוריות/סופרמרקט/חטיפים%2C-מתוקים-ודגני-בוקר/c/A25

I want to get all the li tag with the name class miglog-prod miglog-sellingmethod-by_unit

like this:

<li class="miglog-prod miglog-sellingmethod-by_unit" data-product-name="צדפים דגנים בטעם שוקולד" data-entry-number=""
data-product-box
data-product-replace=""
data-product-price="12.9"
data-product-purchasable="false"
data-food="true"
data-selling-method="BY_UNIT"
data-product-code="P_7296073442226">

I tried:

shufersal = "https://www.shufersal.co.il/online/he/קטגוריות/סופרמרקט/חטיפים%2C-מתוקים-ודגני-בוקר/c/A25"

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
    
driver = webdriver.Chrome(ChromeDriverManager().install())
    
import time
driver.get(shufersal)
for i in range(0,5): # here you will need to tune to see exactly how many scrolls you need
    driver.execute_script('window.scrollBy(0, 400)')
    time.sleep(1)
products = driver.find_elements_by_class_name("miglog-prod miglog-sellingmethod-by_unit")
    
print(products)

but I don't find any product:

[]
selenium-webdriver web-scraping selenium beautifulsoup python
1 Answers

A couple of things you can do:

from selenium.webdriver.common.by import By

products = driver.find_element_by_class_name(
    By.CSS_SELECTOR,
    ".miglog-prod.miglog-sellingmethod-by_unit"
)

or,

products = driver.find_elements_by_css_selector(
    "li.miglog-prod.miglog-sellingmethod-by_unit"
)

Note that in the first method, I am passing just the class names, but in second one I am passing even the element (li).

(Edit: Ok, I checked on REPL, I get elements even if I don't pass li so I am not sure about the behaviour here. Btw, I tested this for Firefox.)

From what I read, you dont get products is because find_elements_by_class_name needs only one class name and you have passed two..

user_42600950
Related