Selenium does not print data from site

Viewed 88

Hey I am a beginner and im trying to get reviews from a site, I have this code:

from selenium import webdriver
chrome_path = r"C:\Users\AV\AppData\Local\Programs\Python\Python39\Scripts\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)

driver.get("https://oldnavy.gapcanada.ca/browse/product.do?pid=676294003&cid=1178957&pcid=1031103&vid=1&grid=pds_5_331_1#pdp-page-content")

review = driver.find_elements_by_class_name("pr-rd-description-text")
for post in review:
    print(post.text)

but it doesn't print anything

this is the HTML source: <p class="pr-rd-description-text">Like the color and price</p>

please tell me where im wrong?

2 Answers

Your code is okay. All you need to do is to scroll down to let the elements visible to Selenium.

Code :

driver.execute_script("window.scrollTo(0, Y)") 

Where You can put any number for Y i.e (100, 200, or so on..)

and You must have seen this button below :

enter image description here

Now we will have to click on down arrow button :

Code :

WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.promoDrawer__handlebar__icon"))).click()

This should work.

Code together :

driver.get("https://oldnavy.gapcanada.ca/browse/product.do?pid=676294003&cid=1178957&pcid=1031103&vid=1&grid=pds_5_331_1#pdp-page-content")
driver.execute_script("window.scrollTo(0, 1000)")
sleep(5)
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.promoDrawer__handlebar__icon"))).click()

review = driver.find_elements_by_class_name("pr-rd-description-text")
for post in review:
    print(post.text)

O/P :

Like the color and price The teal hoodie I received is not the same teal color pictured. It's more a dull darker teal so that was disappointing. I also purchased the sleeveless version which was actual teal color shown, so not sure why this one didn't match.

My son loves this pullover— he even bought the white one My teenage son is really into wearing hoodies, so when he saw this shirt sleeve hoodie he loved it. We purchased a black one and a White one. Feels great and fits great. Enough said. My husband loves short sleeve sweatshirts for the gym. I bought him 2 & he wears them all the time Great fabric

Process finished with exit code 0

You can change your code to this

import time
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains

chrome_path = r"C:\Users\AV\AppData\Local\Programs\Python\Python39\Scripts\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
actions = ActionChains(driver)
waiter = WebDriverWait(driver, 20)

driver.get("https://oldnavy.gapcanada.ca/browse/product.do?pid=676294003&cid=1178957&pcid=1031103&vid=1&grid=pds_5_331_1#pdp-page-content")
driver.implicitly_wait(100)
title = waiter.until(EC.title_contains("Dynamic Fleece"))
time.sleep(5)
element = waiter.until(EC.presence_of_element_located((By.CLASS_NAME, "promoDrawer__handlebar__icon")))
button = driver.find_element_by_class_name("promoDrawer__handlebar__icon").click()
time.sleep(3)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight)")
time.sleep(3)
elmnt = waiter.until(EC.presence_of_all_elements_located((By.CLASS_NAME, "pr-rd-description-text")))
review = driver.find_elements_by_class_name("pr-rd-description-text")
print(review)
for post in review:
    actions.move_to_element(post).perform()
    time.sleep(3)
    print(post.text)

Basically the problem was that the comments would only load on the site when you would scroll till the bottom most of the site
So it was important to scroll down to load comment and get their data.
I have also added some conditions and waiters to make the code run better.
Also added a button click to remove the slide popup that the site shows. :)

Related