How do I find a specific parent/ancestor of an element when using Selenium and Python?

user_11626198 0
Viewed 167

I am currently working on a project that scrapes content from a dynamically generated page and records it.

The below is an example of the structure I am pulling from.

https://dutchie.com/embedded-menu/revolutionary-clinics-somerville/menu

I am able to scroll down the page and locate the specific details from each product card, using: cards = driver.find_elements_by_css_selector("div[class^='product-card__Content']")

From each card, I am able to pull the title, price and other variables and store them as needed.

The problem is that I want to append to these the "category" which happens to appear as a header above each grouping of items, example from above being flower followed by prerolls. As the page scrolls down a new one of these is displayed followed by the related item.

I have been able to access the first instance of this, but no matter where I put the following code in my loop it never pulls the updated value, only the initial.

category = driver.find_element_by_css_selector("div[class^='products-grid__ProductGroupTitle']").text

I tried to figure a way to pull this as the parent/ancestor of the current card, but this concept is still new to me and outside my understanding of how I would solve it. Is locating the "nearest" ProductGroup title the correct approach? If so how would I do so dynamically as the page scrolls?

Thanks!

web-scraping selenium python
2 Answers

To grab all the product group names, you can use a CSS selector

div[class^='products-grid__ProductGroupTitle']

NOTE: ^= in a CSS selector means starts with

From there we can insert the product group into an XPath and find all details for each product under that group.

# loop through all the product names
for product_group_name in driver.find_elements_by_css_selector("div[class^='products-grid__ProductGroupTitle']")
    # loop through each product card
    for product in driver.find_elements_by_xpath("//div[starts-with(@class,'products-grid__ProductGroup')][./div[starts-with(@class,'products-grid__ProductGroupTitle')][text()='" + product_group_name.text + "']]//div[starts-with(@class,'consumer-product-card__InViewContainer')]")
        # get individual product info
        brand = product.find_element_by_css_selector("div[class^='product-information__Brand']")

        # if you use an XPath, make sure you include a dot (.) at the start of the locator
        # brand example using XPath
        brand = product.find_element_by_xpath(".//div[starts-with(@class,'product-information__Brand')]")

        title = product.find_element_by_css_selector("div[class^='product-information__TitleContainer']:not(.mobile-and-card)")
        # ... and so on

The problem is... that page is broken so no code will likely work until it's fixed. If you look in the dev console, you will see repeated error messages.

[mobx.array] Attempt to read an array index (0) that is out of bounds (0). Please check length first. Out of bound indices will not be tracked by MobX

Every time that message fires, the page seems to lose track of elements on the page.

user_23867740

You can use this xpath -

//div[text()="Flower"]/following-sibling::div/div

Just change the text "Flower" with your other categories and you will find all the cards under certain category.

user_98941420
Related