Selenium: Select and iterate through direct <li> child elements of a <ul> parent using python?

user_14355379 0
Viewed 1086

This is the part of the website I want to scrape.

This is the part of the website I want to scrape.

This is my python selenium code:

className = "_29hl"
try:
    search = WebDriverWait(driver, 10).until(
        EC.presence_of_all_elements_located((By.CLASS_NAME, className,))
    )

    print( search )      
    print( search[0] )   
    # the two print statement above print two different webElement objects
    # why is that?

    # I want to do something like this...
    lists = search.direct_children
    links = [ each.find_element_by_tag_name('a') for each in lists ]

    for link in links:
        link.click()
        # other codes...

finally:
    driver.quit()

When I select the className from the console, it returns a collection

enter image description here

So I select the first one, and it returns the <ul> and its <li> child elements I want to scrape

enter image description here

How can it be selected and iterate through <li> elements using python selenium? My goal is to select the <a> inside each <li> and make a click on that

selenium python
1 Answers

presence_of_all_elements_located , finds and returns all the element that has the class as specified , so search is a list containing all elements that has the specified class. when you call search[0] it gets the first element from the list.

to get direct child of this element use below locator

lists = search[0].find_elements_by_xpath("./li")

or

 lists = search[0].find_elements(By.XPATH, "./li")

you can use xpath , . means current element which is the ul , and then '/' means direct child

Full code

className = "_29hl"
try:
    search = WebDriverWait(driver, 10).until(
        EC.presence_of_all_elements_located((By.CLASS_NAME, className,))
    )

    print( search )      
    print( search[0] )   
    # the two print statement above print two different webElement objects
    # why is that?

    # I want to do something like this...
    lists = search[0].find_elements_by_xpath("./li")

    links = [ each.find_elements_by_tag_name('a') for each in lists ]

    for link in links:
        link.click()
        # other codes...

finally:
    driver.quit()
user_67936370
Related