Exporting selenium scraped data to CSV file?

user_11524675 0
Viewed 81

In the following script, I scraped the coronavirus data from a table on worldometers.info/coronavirua with selenium.

from time import sleep
from selenium import webdriver

class CoronaBot():
def __init__(self):
    self.driver = webdriver.Chrome()

def scraper(self):
    self.driver.get('https://worldometers.info/coronavirus/')
    main_table = self.driver.find_element_by_xpath('//*[@id="main_table_countries_today"]')
    country = main_table.find_element_by_xpath("//td[contains(., 'Austria')]")
    row = country.find_element_by_xpath("./..")
    data = row.text.split(" ")
    total_cases = data[0]
    new_cases = data[1]
    total_deaths = data[2]
    new_deaths = data[3]
    active_cases = data[4]
    total_recovered = data[5]
    serious_critical = data[6]

The code works fine, I could just print it out like this:

    print("COVID-19 updates in: " + country.text)
    print("Total Cases: " + total_cases)
    ...

However, I want to take the output of the scraped results and place it in a new csv file (the csv file needs to be created when the script is run.)

I tried something stupid like this in panda, but it obviously did not work. Any suggestion?

def create_csv(self):

    collected_data = []

    collected_data.append(output)

    df = pd.DataFrame(collected_data, columns=['total_cases', 'new_cases', 'total_deaths', 
    'new_deaths', 'active_cases', 'total_recovered','serious_critical'])
    df.to_csv('scraped_corona.csv')
selenium csv python
1 Answers

Pandas is a great solution, you were close. In your example, you can just use the scraper function to put data in the data frame right away.

At first, I'd create self.df attribute to store the data frame:

class CoronaBot():
    def __init__(self):
        self.driver = webdriver.Chrome()
        column_names = ['total_cases', 'new_cases', 'total_deaths', 'new_deaths','active_cases', 'total_recovered', 'serious_critical']
        self.df = pd.DataFrame(columns=column_names)

Then, after you collect the data, store it in self.df:

...
print("Total recovered: " + total_recovered)
print("Serious, critical cases: " + serious_critical)

self.df = self.df.append(
    {'total_cases': total_cases,
     'new_cases': new_cases,
     'total_deaths': total_deaths,
     'new_deaths': new_deaths,
     'active_cases': active_cases,
     'total_recovered': total_recovered,
     'serious_critical': serious_critical}, ignore_index=True)

And add a function for exporting:

    def export_to_csv(self):
        self.df.to_csv('scraped_corona.csv')

Now, when I run

c = CoronaBot()
c.scraper()
c.export_to_csv()

I get the .csv file. Hope it helps, good luck!

user_77469920
Related