Scraping one column from a HTML table using BeautifulSoup

Question

I know there n number of questions on BeautifulSoup but after trying a few things I am unable to figure out how to parse the data I need from this HTML table.

My table looks like this:


    
        
            Date
            Open
            High
            Low
            Close*
            Adj Close**
            Volume
        
    
    
        
            Oct 10, 2019
            2,918.55
            2,948.46
            2,917.12
            2,938.13
            2,938.13
            3,217,250,000
        
        
            Oct 09, 2019
            2,911.10
            2,929.32
            2,907.41
            2,919.40
            2,919.40
            2,726,820,000

I want to extract the data from the column "Adj Close". The problem I am having is that all class attributes have the same value.

How do I just extract all the data in the "Adj Close" column?

Here is my code so far:

import pandas as pd
import numpy as np

raw_html = simple_get('https://finance.yahoo.com/quote/%5EGSPC/history?p=%5EGSPC')
soup = BeautifulSoup(raw_html, 'html.parser')
table = soup.find("table", attrs={'class':'W(100%) M(0)'})
stock_history_list=[]

try:
    for row in table.find_all('tr'):
        cols = row.find_all('td')
        print(cols)
        if len(cols) > 0:
            stock_history_list.append(cols[5].text.strip())
except: pass  

stock_history_array = np.asarray(stock_history_list)
df = pd.DataFrame(stock_history_array)

user_7326738 · Answer

You can transform the HTML into a list of dictionaries for quick lookup:

header, *data = [[i.text for i in b.find_all('th' if not b.td else 'td')] for b in d.find_all('tr')]
result = [dict(zip(header, i)) for i in data]
vals = [i['Adj Close**'] for i in result]

Or, using pandas:

import pandas as pd
df = pd.DataFrame(result)
vals = df['Adj Close**']

Output:

0    2,938.13
1    2,919.40
Name: Adj Close**, dtype: object

user_6241235 · Answer

You can use nth-of-type (If you know index then specify direct or I show how to get it based on headers). Using bs4 4.7.1+

from bs4 import BeautifulSoup as bs

html = '''
    
        
            Date
            Open
            High
            Low
            Close*
            Adj Close**
            Volume
        
    
    
        
            Oct 10, 2019
            2,918.55
            2,948.46
            2,917.12
            2,938.13
            2,938.13
            3,217,250,000
        
        
            Oct 09, 2019
            2,911.10
            2,929.32
            2,907.41
            2,919.40
            2,919.40
            2,726,820,000
        
'''
soup = bs(html, 'lxml')
index = [th.text for th in soup.select('[data-test="historical-prices"] th')].index('Adj Close**') + 1
data = [td.text for td in soup.select(f'[data-test="historical-prices"] td:nth-of-type({index})')]
print(data)

Scraping one column from a HTML table using BeautifulSoup

2 Answers