Selenium Scrape not reading all elements

Question

I am trying to scrape data from the following site. I was able to click on load more yet the code doesn't catch most of the elements and I do not really know what to do.

url = 'https://www.carrefouregypt.com/mafegy/en/c/FEGY1701230'
products = []

options = Options()
driver = webdriver.Chrome(options = options)
driver.get(url)
time.sleep(8)

#click on load more
while True:
    try:
        btn_class = 'css-1n3fqy0'
        btn = driver.find_element(By.CLASS_NAME , btn_class)
        btn.click()
        driver.implicitly_wait(10)

    except NoSuchElementException:
        break

driver.execute_script("window.scrollTo(0,document.body.scrollHeight)")
time.sleep(8)

user_19475185 · Answer

The following code will click that button until it cannot locate it, and exit gracefully:

from selenium.common.exceptions import NoSuchElementException, TimeoutException
from selenium import webdriver
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.firefox.options import Options as Firefox_Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support import expected_conditions as EC
import time as t

firefox_options = Firefox_Options()

firefox_options.add_argument("--width=1280")
firefox_options.add_argument("--height=720")
# firefox_options.headless = True
firefox_options.set_preference("general.useragent.override", "Mozilla/5.0 (Linux; Android 7.0; SM-A310F Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.91 Mobile Safari/537.36 OPR/42.7.2246.114996")

driverService = Service('chromedriver/geckodriver')

browser = webdriver.Firefox(service=driverService, options=firefox_options)

url = 'https://www.carrefouregypt.com/mafegy/en/c/FEGY1701230'

browser.get(url) 
t.sleep(5) 
while True:
    try:
        load_more_button = WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.XPATH,'//button[text()="Load More"]')))
        browser.execute_script('window.scrollBy(0, 100);')
        load_more_button.click()
        print('clicked')
        t.sleep(3)       
    except TimeoutException:
        print('all elements loaded in page')
        break

It's using Firefox, on a linux setup (for some reasons Chrome was temperamental on this one). You just have to observe the imports, and the code after defining the browser/driver. Selenium documentation: https://www.selenium.dev/documentation/

Selenium Scrape not reading all elements

1 Answers