Selenium problem [don't show up error](download few items)

Question

I'm in need of a solution to my code, I tried to web scraping a dynamic web page call easy.cl and just get 4 items and sometimes none (only when I download title, cant download price because don't show anything). Well, anyhow, I need a guide of where is my error, because Selenium don't show me any in my result (Sublime text3). Also, easy.cl is dynamic going with a button to show more info products. Finally, I'm thinking in a scroll solution but can't tell, What would you do in my position? any tip to try to find a solution?

in advanced, thanks.

import random
from time import sleep
from selenium import webdriver
import pandas as pd
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)

#baseUrl = 'https://www.easy.cl'

driver.get('https://www.easy.cl/tienda/categoria/ceramicas?cur_page=1&cur_view=grid&bc_links=pisos-y-muros&bc_names=Pisos')

#boton = driver.find_element_by_xpath('//button[@class="primary_plp load-more-products"]')
inicio = []
for i in range(5):
              try:
                            boton = WebDriverWait(driver, 10).until(
          EC.presence_of_element_located((By.XPATH, '//button[@class="primary_plp load-more-products"]')))
                            boton.click()
                            WebDriverWait(driver, 10).until(
          EC.presence_of_element_located((By.XPATH,'//a[@class="product_image"]')))
              except:
                            break

    # Espero que carguen los productos...
links_productos = WebDriverWait(driver,10).until(EC.presence_of_all_elements_located((By.XPATH,'//a[@class="product_image"]')))
# obtengo los links de los detalles de los productos
links_pagina=[]
for tag_a in links_productos:
              links_pagina.append(tag_a.get_attribute("href"))

for link in links_pagina:
              try:
                            driver.get(link)
                            titulo = driver.find_element(By.XPATH, '//h1[@class="product-details__title"]').text
                            #if driver.find_element(By.XPATH, '//span[@class="priceNumber priceNumberAlignRight"]'):
                            #            precio = find_element_by_xpath('//span[@class="priceNumber priceNumberAlignRight"]').text
                            #else:
                            #            precio = "Sin precio M2"

                            print(titulo)
                            #print(precio)
                                          
              except:
                            break

user_3485434 · Answer

The load more products button appears on the bottom of the page, out of the visible screen, so possibly after the element is presented (loaded) you need to scroll to that element before clicking it

from selenium.webdriver.common.action_chains import ActionChains


boton = WebDriverWait(driver, 10).until(
          EC.presence_of_element_located((By.XPATH, '//button[@class="primary_plp load-more-products"]')))
actions = ActionChains(driver)
actions.move_to_element(boton).perform()
boton.click()

But the main issue here, as I see, is

links_productos = WebDriverWait(driver,10).until(EC.presence_of_all_elements_located((By.XPATH,'//a[@class="product_image"]')))

You assume that this will give you all the elements that can be presented on that page matching that locator, but actually this expected condition is waiting for at least 1 element matching this condition and once it catches it - it returns a list of web elements matching that locator it found.
I see no expected condition in Python like we have in Java like this ExpectedConditions.numberOfElementsToBe or this ExpectedConditions.numberOfElementsToBeMoreThan that is actually waiting for desired amount of elements located by passed locator. So, what I can advise you here, is to add a delay of 1-2 seconds after clicking the boton.click() before reading the links_productos. This should resolve your problem.

Selenium problem [don't show up error](download few items)

1 Answers