extracting content from 1 column of table with beautiful soup

user_16906207 0
Viewed 46

I'm new to web scraping, and I'm trying to pull all the names of all the journals from this website: https://ideas.repec.org/top/top.journals.simple.html.

Here's my attempt so far (following the tutorial here https://www.pluralsight.com/guides/extracting-data-html-beautifulsoup):

import requests
from bs4 import BeautifulSoup

URL = "https://ideas.repec.org/top/top.journals.simple.html"
html_content = requests.get(URL).text
soup = BeautifulSoup(html_content, "lxml"

journal_list = soup.find("table", attrs={"class": "toplist"})
journal_list_data = journal_list.tbody.find_all("tr")

headings = []

for td in journal_list_data[0].find_all("td"):
     headings.append(td.b.text.replace('\n', '').strip())

print(headings)

This is just to get a list of the table headings, and then I'll try to pull all the journal names from the 'Journal' column, but I'm getting an AttributeError basically saying that journal_list.tbody is NoneType, and when I check journal_list.attrs it only gives {'class': 'toplist'), even though the HTML on the page definitely has a tbody attribute.

What am I doing wrong/are there other, better ways to do this?

Thanks!

beautifulsoup html python
1 Answers

What happens?

First at all take a look into your soup, there is your truth - There is no

<tbody> and also no <b> to get information from. Thats why you end in an error.

Try this instead

import requests
from bs4 import BeautifulSoup

URL = "https://ideas.repec.org/top/top.journals.simple.html"
html_content = requests.get(URL).text
soup = BeautifulSoup(html_content, "lxml")

journal_list = soup.find("table", attrs={"class": "toplist"})
journal_list_data = journal_list.find_all("tr")

headings = []

for td in journal_list_data[0].find_all("td"):
     headings.append(td.text.replace('\n', '').strip())

print(headings)

Alternative get_text()

You can also use get_text() to grab and strip the text of an element:

headings.append(td.get_text(strip=True))

Solution - One Column of table

To get only the text from one column e.g. the journal you can do the following:

[journal.find_all("td")[1].get_text(strip=True) for journal in journal_list_data[1:]]
user_144608240
Related