click link automatically and scraping

user_11455772 0
Viewed 61

I try to extract the all the products data from this page:

https://www.shufersal.co.il/online/he/קטגוריות/סופרמרקט/חטיפים%2C-מתוקים-ודגני-בוקר/c/A25

I tried

shufersal = "https://www.shufersal.co.il/online/he/%D7%A7%D7%98%D7%92%D7%95%D7%A8%D7%99%D7%95%D7%AA/%D7%A1%D7%95%D7%A4%D7%A8%D7%9E%D7%A8%D7%A7%D7%98/%D7%97%D7%98%D7%99%D7%A4%D7%99%D7%9D%2C-%D7%9E%D7%AA%D7%95%D7%A7%D7%99%D7%9D-%D7%95%D7%93%D7%92%D7%A0%D7%99-%D7%91%D7%95%D7%A7%D7%A8/c/A25"
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(ChromeDriverManager().install())

import time
driver.get(shufersal)
products = driver.find_elements_by_css_selector("li.miglog-prod.miglog-sellingmethod-by_unit"
)

the problem is that the product details is showed only when I click the product:

enter image description here

enter image description here

There any option to click all links automatically and scraping the open windows?

web-scraping selenium python
1 Answers

What you want can be achieved, but is a significantly more time consuming processes.

You'll have to:

identify the elements that you need to click in the page (they'll probably have the same class) and then select them all:

butons_to_click = driver.find_elements_by_css_selector({RELEVANT SELECTOR HERE})

Then you should loop through all clickable elements, click on them, wait for the pop up to load, scrape the data, close the popup:

scraped_list = []
for button_instance in buttons_to_click:
    button_instance.click()
    #scrape the information you need here and append to scraped_list

    #find close popup button
    driver.find_element_by_xpath({XPATH TO ELEMENT}).click()

For this to work properly it is important to setup selenium's implicitly wait parameter. What that does is that if it doesn't find the required element it'll wait for X seconds until it is loaded, if X passes then it'll throw an error (you can handle the error in your code if it is expected).

In your case, you'll need the wait time because after you click on the product to display the popup, the information might take some seconds to load, in case you don't set implicitly wait, your script will exit with an element not found error. More information on selenium's wait parameters can be found here: https://selenium-python.readthedocs.io/waits.html

#put this line immediatelly after creating the driver object
driver.implicitly_wait(10) # seconds

***Suggestion: I suggest you to always use Xpath when looking up elements, its syntax can actually emulate all other selenium selectors, is faster and will make an easier transition to C compiled html parsers (which you'll need in case you scale your scraper - recommend lxml a python package that uses a compiled parser)

user_115848390
Related