Selenium and Pandas to scrape table in page

user_1708942 0
Viewed 178

I am trying to load the table available in 'https://umich-biostatistics.shinyapps.io/covid19/' and navigating to 'Metrics' section in page into a dataframe. Becuase the page loads the data after opening the page, I have tried to use selenium. Could someone help in figuring out my mistake?

import time
from selenium import webdriver
import pandas as pd

chrome_path = r"C:\\Selenium\\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
url = 'https://umich-biostatistics.shinyapps.io/covid19/'

page = driver.get(url)
time.sleep(10)

df = pd.read_html(driver.page_source)[0]
print(df.head())

By running the above code, I am getting the below error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-10-4781af4c4464> in <module>
     10 time.sleep(10)
     11 
---> 12 df = pd.read_html(driver.page_source)[0]
     13 print(df.head())

~\anaconda3\lib\site-packages\pandas\io\html.py in read_html(io, match, flavor, header, index_col, skiprows, attrs, parse_dates, thousands, encoding, decimal, converters, na_values, keep_default_na, displayed_only)
   1098         na_values=na_values,
   1099         keep_default_na=keep_default_na,
-> 1100         displayed_only=displayed_only,
   1101     )

~\anaconda3\lib\site-packages\pandas\io\html.py in _parse(flavor, io, match, attrs, encoding, displayed_only, **kwargs)
    913             break
    914     else:
--> 915         raise retained
    916 
    917     ret = []

~\anaconda3\lib\site-packages\pandas\io\html.py in _parse(flavor, io, match, attrs, encoding, displayed_only, **kwargs)
    893 
    894         try:
--> 895             tables = p.parse_tables()
    896         except ValueError as caught:
    897             # if `io` is an io-like object, check if it's seekable

~\anaconda3\lib\site-packages\pandas\io\html.py in parse_tables(self)
    211         list of parsed (header, body, footer) tuples from tables.
    212         """
--> 213         tables = self._parse_tables(self._build_doc(), self.match, self.attrs)
    214         return (self._parse_thead_tbody_tfoot(table) for table in tables)
    215 

~\anaconda3\lib\site-packages\pandas\io\html.py in _parse_tables(self, doc, match, attrs)
    543 
    544         if not tables:
--> 545             raise ValueError("No tables found")
    546 
    547         result = []

ValueError: No tables found
pandas selenium python
1 Answers

Use -

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd

chrome_path = r"C:\\Selenium\\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
url = 'https://umich-biostatistics.shinyapps.io/covid19/'

page = driver.get(url)

try:
    element = WebDriverWait(driver, 1000).until(
        EC.presence_of_element_located((By.CLASS_NAME, "gt_table"))
    )
    df = pd.read_html(driver.page_source)[0]
finally:
    driver.quit()
    print(df.head())

Output

             Day   Date  Deaths   Cases    Tests    TPR  Vaccines
0          Today  05/01    3685  392576  1504698  26.1%   1826396
1      Yesterday  04/30    3525  402014  1804954  22.3%   2744456
2   One week ago  04/24    2761  348996  1402367  24.9%   2536585
3  One month ago  04/01     468   81398  1046605   7.8%   3671242

Explanation

time.sleep() is a very crude way of waiting for the webpage to load. Your guess is as good as mine on how long it will take for an element to load.

Now, in your case, the website loads fully, but since this is a shiny app, the table loads asynchronously much after the webpage loads completely.

For this reason, time.sleep(10) is a hit/miss as even of the webpage is loaded, the table might not be retrieved completely.

The ideal implementation in this case, will be to use WebDriverWait with a hard timeout (you can't wait anymore than this!) so that the program moves the moment the table loads.

In your webpage the table had a style tag of gt_table based on which I was able to pull it. You ideally want to have an ID element so you can pull by By.ID or better way will be By.XPATH if you don't expect the webpage to change much.

user_40980130
Related