How to retrieve and store 2nd and 3rd row elements in a dataframe

Question

I am new to Pandas and Webscraping and BeautifulSoup in Python.

As I was learning to do some basic webscraping in Python by using requests and BeautifulSoup to scrape a webpage, I am confused with the task of assigning the 2nd and 3rd elements of an html table into a pandas dataframe.

Suppose I have this table:

Here is my code so far:

import pandas as pd
from bs4 import BeautifulSoup
import requests

html_data = requests.get('https://en.wikipedia.org/wiki/List_of_largest_banks').text
soup = BeautifulSoup(html_data, 'html.parser')
data = pd.DataFrame(columns=["Name", "Market Cap (US$ Billion)"])

for row in soup.find_all('tbody')[3].find_all('tr'): #This line will make sure to get to the third table which is this "By market capitalization" table on the webpage and finding all the rows of this table
  col = row.find_all('td') #This is to target individual column values in a particular row of the table
  for j, cell in enumerate(col):
    #Further code here

As it can be seen, I want to target all the 2nd and 3rd column values of a row and append to the empty dataframe, data, so that data contains the Bank names and market cap values. How can I achieve that kind of functionality?

user_7475838 · Answer

For tables I would suggest pandas:

import pandas as pd
url = 'https://en.wikipedia.org/wiki/List_of_largest_banks'
tables = pd.read_html(url)
df = tables[1]

When you prefer using beautifulsoup, you can try this to accomplish the same:

url = 'https://en.wikipedia.org/wiki/List_of_largest_banks'
r = requests.get(url)
soup = BeautifulSoup(r.content, 'html.parser').find_all('table')
table = soup[1]
table_rows = table.find_all('tr')
table_header = [th.text.strip() for th in table_rows[0].find_all('th')]
table_data = []
for row in table_rows[1:]:
    table_data.append([td.text.strip() for td in row.find_all('td')])
df = pd.DataFrame(table_data, columns=table_header)

When needed, you can set Rank as index with df.set_index('Rank', inplace=True. Image below is the unmodified dataframe.

How to retrieve and store 2nd and 3rd row elements in a dataframe

1 Answers