I'm trying to scrape covid data from an online table (can be found by visiting this site, and clicking the pink "State Data" button on the far-right end of the sub-header). I used a combination of selenium (click the button, then store the source html using
driver.page_source()) and BeautifulSoup to parse through the page's html. On my first run, I was able successfully locate the table, loop through the table's rows, and store the rows in a pandas.DataFrame no problem (using this method). However when attempting it again later on, I'm not able to locate all the table's rows like I did the first time.
Here's my code:
from selenium import webdriver
from bs4 import BeautifulSoup
import pandas as pd
driver = webdriver.Chrome(executable_path = 'C:/Work/chromedriver.exe')
driver.get('https://www.mohfw.gov.in/')
# click state data
driver.find_element_by_xpath('//*[@id="site-dashboard"]/div/div/div[1]/div[3]/ul/li/a[2]/img').click()
page_html = driver.page_source
driver.quit()
##
soup = BeautifulSoup(page_html, "lxml")
table = soup.find('table', attrs={'class':'statetable table table-striped'})
table_rows = table.find_all('tr')
# append each row to a list
l = []
for tr in table_rows:
td = tr.find_all('td')
row = [tr.text for tr in td]
l.append(row)
# get rid of the rows that have less than 7 entries
# (each entry represents a column. The last row has 7 because it's a total and wasn't given a row ID)
l2 = [i for i in l if len(i)>=7]
for count, element in enumerate(l2):
if len(element) == 7:
element.insert(0, count+1)
# Convert to DataFrame
pd.DataFrame(l2, columns=["S. No.",
"Name of State / UT",
"Total Active Cases",
"Change in Total Active Cases Since Yesterday",
"Cumulative Cured/Discharged/Migrated",
"Change in Cumulative Cured/Discharged/Migrated Since Yesterday",
"Cumulative Deaths",
"Change in Cumulative Deaths Since Yesterday"]).replace(r'\xa0 ', pd.NA, regex=True)
From what I read online, there may be broken HTML on the page. When I threw the output of the table variable into sublime text and set the syntax to HTML, I noticed that the section of HTML I needed to parse was greyed out (vs. the rest of the colorful text at the top, only of which was being returned by find_all()). Something is telling me that the <!--<tbody> tag is what's greying out the section I need to parse, but my attempts at removing it have been unsuccessful. I know very little about HTML so maybe a more experienced eye could quickly deduce what's going on.