I am getting an error while saving data to csv

user8h671 0
Viewed 42

I am trying to save data from multiple website to csv using selenium

driver = webdriver.Chrome(executable_path='C:/Users/trive/Downloads/chromedriver_win32/chromedriver.exe')
templist = []

for i in range(len(data)):
    driver.get(data._get_value(i,'URL'))
    
    try:
        title= driver.find_element(By.CLASS_NAME, "amp-wp-title").text  
        content= driver.find_element(By.CLASS_NAME, "amp-wp-article-content").text 
      
        Table_dict={ 'Title': title,
                    'Content':content}
      
        templist.append(Table_dict) 
        df = pd.DataFrame(templist)
   

    except NoSuchElementException: 
        continue


# saving the dataframe to a csv
df.to_csv('S:/DEADPOOL/black coffr assignment/table.csv')
driver.close()

I am getting the following error message:

 NameError                                 Traceback (most recent call last)
 <ipython-input-18-8534ce45b80d> in <module>
 ---> 28 print(df)

 NameError: name 'df' is not defined

I think it is to do with df only existing in the for loop and not outside but i don't know how to solve this.

selenium pandas python
1 Answers

In your for loop you have a try/except clause, where the except clause just does continue. I suspect there's an error somewhere in the try section preventing the creation of the dataframe df

In other words, this line

df = pd.DataFrame(templist)

probably never gets executed as it is skipped due to an error that is always being caught by

except NoSuchElementException

Another thing to note is that this line

df.to_csv('S:/DEADPOOL/black coffr assignment/table.csv')

is being run outside the loop, which means it will only be executed once after the loop ends, so only the last iteration of df = pd.DataFrame(templist) would matter if it actually did run without errors.

anthony22610
Related