Python:- Extracting the column headers from rows[0] enclosed in a <th> tag. Remove the unicode symbol and the link number after the planet name

user_14791248 0
Viewed 80

I am trying to iterate over all the column headers of row[0] then remove the unicode symbol and the link number after the planet name. Right now my code looks like this:-

URL_solar_system = "https://en.wikipedia.org/wiki/List_of_gravitationally_rounded_objects_of_the_Solar_System"

from bs4 import BeautifulSoup

import requests

html_content = requests.get(URL_solar_system).text

soup_solar = BeautifulSoup(html_content, "lxml")

tables = soup_solar.find_all('table', attrs={'class': 'wikitable'})

planets = tables[2]

rows = planets.find_all('tr')

headers = [th.text.strip() for th in rows[0].find_all('th') if th.get_text().strip() != '' ] 

print('headers: {}'.format(headers))

OUTPUT

headers: ['*Mercury[6][7]', '*Venus[8][9]', '*Earth[10][11]', '*Mars[12][13]', '°Jupiter[14][15]', '°Saturn[16][17]', '‡Uranus[18][19]', '‡Neptune[20][21]']

I am trying to achieve the output as shown below. Desired Output:

headers: ['Mercury', 'Venus', 'Earth', 'Mars', 'Jupiter', 'Saturn', 'Uranus', 'Neptune']

Hint for writing the code:-

  1. headers = []
  2. for loop that will loop through row[0] with a find all on 'th'
  3. first_variable = uses your iterator (let's say we call it i in the for loop definition) and uses the text method on it
  4. second_variable = find this value '[' from the first variable that has the text method redefine the first variable by indexing the second element to the second variable element you defined.
  5. if statement for that label where is appends the first variable to the headers list you created.
beautifulsoup python-3.x python
2 Answers

Add

import re # in the begining of your module

then just add to the end of your code the following line

headers = re.findall(r'(\w+)\[\d+]',''.join(headers))

Your final code

import re

URL_solar_system = "https://en.wikipedia.org/wiki/List_of_gravitationally_rounded_objects_of_the_Solar_System"

from bs4 import BeautifulSoup

import requests

html_content = requests.get(URL_solar_system).text

soup_solar = BeautifulSoup(html_content, "lxml")

tables = soup_solar.find_all('table', attrs={'class': 'wikitable'})

planets = tables[2]

rows = planets.find_all('tr')

headers = [th.text.strip() for th in rows[0].find_all('th') if th.get_text().strip() != '' ] 

headers = re.findall(r'(\w+)\[\d+]',''.join(headers))

print('headers: {}'.format(headers))
user_82868740

The header text are href's under the fourth table. Try using a CSS Selector that selects the fourth table: table:nth-of-type(4), and then use th > a which selects the a tags (the headers) under the th tags.

import requests
from bs4 import BeautifulSoup

URL = "https://en.wikipedia.org/wiki/List_of_gravitationally_rounded_objects_of_the_Solar_System"
soup = BeautifulSoup(requests.get(URL).content, "html.parser")

print([tag.text for tag in soup.select("table:nth-of-type(4) th > a")])

Output:

['Mercury', 'Venus', 'Earth', 'Mars', 'Jupiter', 'Saturn', 'Uranus', 'Neptune']
user_123497340
Related