Click each search result - iterate over elements found - Selenium Python

user_15662327 0
Viewed 123

I want to write a piece of code in Python that would click every result of a search and within each seperate result download all the files.

I'm trying to write a for loop that would iterate over the search results, however when I run the code it's not doing anything, neither throws an error. I am a newbie to Selenium and would appreciate if you could tell me where the problem lies, if it's the classes I'm using or the code itself.

First I open the search result window:

driver.get("https://dane.gov.pl/pl/dataset?page=1&per_page=100&q=&sort=-date&format%5Bterms%5D=csv&types%5Bterms%5D=file")

Closing a popup window:

WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, "footer-close"))).click()

My for loop:

elements = driver.find_elements_by_class_name("result-item ng-star-inserted")
count = 0 

for e in elements:
    e.click()
    for file in driver.find_element_by_class_name("ng-star-inserted"):
        file.click()
        count+=1

I'm not entirely sure if I'm using the correct class names, since the code doesn't open any link but there's no error.

The code can be easily run if selenium and drivers installed.

selenium python
1 Answers

First of all since the locator has multiple class names you should use css selector or XPath, not by_class_name.
Otherwise this will give you 0 results, this is what you have currently

elements = driver.find_elements_by_css_selector(".result-item.ng-star-inserted")
count = 0 

for e in elements:
    e.click()
    for file in driver.find_element_by_class_name("ng-star-inserted"):
        file.click()
        count+=1

After clicking the e element another page is opened.
This will make all the rest of elements stale elements so you wil not be able to use them.
Also, driver.find_element_by_class_name("ng-star-inserted") seems to be wrong, you are using a wrong locator here.
The correct locator there is //div[@class='result-item ng-star-inserted']//h2//a xpath so the code will be

elements = driver.find_elements_by_css_selector(".result-item.ng-star-inserted")
count = 0 

for e in elements:
    e.click()
    for file in driver.find_element_by_xpath("//div[@class='result-item ng-star-inserted']//h2//a"):
        file.click()
        count+=1

But you still have to solve the problem with stale elements - search results from the first page.
I would advise doing this as following:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains

actions = ActionChains(driver)
wait = WebDriverWait(driver, 20)

#get the amount of search results
elements = driver.find_elements_by_css_selector(".result-item.ng-star-inserted")
count = 0 

#iterate for all the result
for i in range(1,len(elements)+1):
    #get the n-th search result link    
    wait.until(EC.visibility_of_element_located((By.XPATH, "(//div[@class='result-item ng-star-inserted'])[" + str(i) + "]")))).click()
        #inside the page get the links to be clicked
    wait.until(EC.visibility_of_element_located((By.XPATH, "//div[@class='result-item ng-star-inserted']//h2//a")))
    files = driver.find_elements_by_xpath("//div[@class='result-item ng-star-inserted']//h2//a")
    for j in range(1,len(files)+1):
        time.sleep(10)
        file = driver.find_element_by_xpath("(//div[@class='result-item ng-star-inserted']//h2//a)[" + str(j) + "]")
        actions.move_to_element(file).perform()
        time.sleep(0.5)
        file.click()
        time.sleep(2)
        count+=1
    #go back to results page
    driver.execute_script("window.history.go(-1)")
user_34854340
Related