getting selenium html table into pandas dataframe with beautifulsoup

user_12596414 0
Viewed 839

I have successfully used selenium to scrape an html table which required a button click prior to scraping.

So selenium works, and successfully stores the html table as a variable ' r '.

I'm stumbling however getting this parsed into a pandas dataframe.

Like the URL, the dataframe should have 5 columns and about 30 rows.

Can anyone see where this is going wrong?

import pandas as pd
from selenium import webdriver
from bs4 import BeautifulSoup

browser = webdriver.Firefox(executable_path=r'/Users/computer_name/Documents/python/web_drivers/geckodriver')
browser.get('https://www.investing.com/equities/exxon-mobil-income-statement')
linkElem = browser.find_element_by_link_text('Annual')
linkElem.click()
r = browser.find_element_by_css_selector("#rrtable > table").get_attribute('innerHTML')
browser.quit()

soup = BeautifulSoup(r, 'html.parser')

df = pd.DataFrame(soup)
print(df)

Many thanks

pandas selenium beautifulsoup
1 Answers

Once you get the soup element then use pd.read_html() You need to use outerHTML instead of innerHTML

r = browser.find_element_by_css_selector("#rrtable > table").get_attribute('outerHTML')
browser.quit()
soup = BeautifulSoup(r, 'html.parser')
df = pd.read_html(str(soup))[0]
print(df)

Output:

                                     Period Ending:  ...                                          201631/12
0                                       Total Revenue  ...                                             200628
1   Revenue 255583 279332 237162 200628  Other Rev...  ...  Revenue 255583 279332 237162 200628  Other Rev...
2                                             Revenue  ...                                             200628
3                                Other Revenue, Total  ...                                                  -
4                              Cost of Revenue, Total  ...                                             136098
5                                        Gross Profit  ...                                              64530
6                            Total Operating Expenses  ...                                             199692
7   Selling/General/Admin. Expenses, Total 41923 4...  ...  Selling/General/Admin. Expenses, Total 41923 4...
8              Selling/General/Admin. Expenses, Total  ...                                              39819
9                              Research & Development  ...                                               1467
10                        Depreciation / Amortization  ...                                              22308
11          Interest Expense (Income) - Net Operating  ...                                                  -
12                           Unusual Expense (Income)  ...                                                  -
13                    Other Operating Expenses, Total  ...                                                  -
14                                   Operating Income  ...                                                936
15       Interest Income (Expense), Net Non-Operating  ...                                               4353
16                      Gain (Loss) on Sale of Assets  ...                                                  -
17                                         Other, Net  ...                                               2680
18                            Net Income Before Taxes  ...                                               7969
19                         Provision for Income Taxes  ...                                               -406
20                             Net Income After Taxes  ...                                               8375
21                                  Minority Interest  ...                                               -535
22                               Equity In Affiliates  ...                                                  -
23                                U.S GAAP Adjustment  ...                                                  -
24              Net Income Before Extraordinary Items  ...                                               7840
25                          Total Extraordinary Items  ...                                                  -
26                                         Net Income  ...                                               7840
27                    Total Adjustments to Net Income  ...                                                  -
28  Income Available to Common Excluding Extraordi...  ...                                               7840
29                                Dilution Adjustment  ...                                                  -
30                                 Diluted Net Income  ...                                               7840
31                    Diluted Weighted Average Shares  ...                                               4177
32          Diluted EPS Excluding Extraordinary Items  ...                                               1.88
33                   DPS - Common Stock Primary Issue  ...                                               2.98
34                             Diluted Normalized EPS  ...                                               1.88
user_108856840
Related