iterate over a list in Selenium and saving the results into a dataframe

user_16651351 0
Viewed 195

I'm trying to iterate over a list, search on a webpage via selenium and store the results in a df. How can store the loop results from each list item into a df?

from selenium.webdriver.common.keys import Keys
import pandas as pd
import numpy as np


url  = 'https://au.finance.yahoo.com/australia/'
driver_path = 'chromedriver.exe'
browser = Chrome(executable_path= driver_path)
loop_search = browser.find_element_by_id('yfin-usr-qry')

search_companies = ['Commonwealth Bank','Rio Tinto','Wesfarmers']


for i in search_companies:
    loop_search.send_keys(i)
    browser.find_element_by_id('search-button').click()
    comp = browser.find_element_by_id('quote-header-info').text
    df3 = [comp]```

Still fairly new to Python! Thank you!
jupyter pandas selenium python
1 Answers

If you just run your code and do print(comp)

you'd see the below error:

selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
(Session info: chrome=99.0.4844.74)

so before saving it to the DF, we need to resolve this:

that can be fixed by redefining the web element like this in the loop:

loop_search = wait.until(EC.visibility_of_element_located((By.ID, "yfin-usr-qry")))

Full code to save it to DF:

driver_path = 'chromedriver.exe'
browser = Chrome(executable_path= driver_path)
wait = WebDriverWait(driver, 20)
url  = 'https://au.finance.yahoo.com/australia/'
driver.get(url)

search_companies = ['Commonwealth Bank','Rio Tinto','Wesfarmers']

company_details_lst = []
for i in search_companies:
    time.sleep(2)
    loop_search = wait.until(EC.visibility_of_element_located((By.ID, "yfin-usr-qry")))
    loop_search.send_keys(i)
    time.sleep(2)
    wait.until(EC.element_to_be_clickable((By.ID, "search-button"))).click()
    time.sleep(2)
    comp = wait.until(EC.element_to_be_clickable((By.ID, "quote-header-info"))).text
    company_details_lst.append(comp)
    #print(comp)

data = {
         'Details': company_details_lst
        }

df = pd.DataFrame.from_dict(data)
df.to_csv('output.csv', index = 0

Imports:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

after running your code you should see a csv file in your project folder with name as output.csv

and the internal content would be:

Output

user_94440730
Related