No return while extracting span class text using BeauitfulSoup

user_4459644 0
Viewed 125

I'm trying to grab the headlines from the NY Times front page (www.nytimes.com). Process finishes without printing text (or anything) from the soup.find_all function.

I've played around with the syntax, changing it from soup.find_all(class_="blancedHeadline") to soup.find_all("span", {"class" : "blancedHeadline") even adding attrs= right before the class distinction.

Here's the code I have and after trying to figure out what I'm doing wrong for a while, I don't know what's causing the issue:

import requests
from bs4 import BeautifulSoup

base_url = 'https://www.nytimes.com/'
r = requests.get(base_url)
soup = BeautifulSoup(r.text, 'lxml')

headline_text = soup.find_all('span', {'class':'balancedHeadline'})

for headline in headline_text:
    print(headline)
web-scraping beautifulsoup python-3.x html python
2 Answers

First of all, why you are not able get an output using "blancedHeadline" class is because the page is partially rendered using javascript. You might be able to see it in the "inspect" tool of your browser. But, if you go and check the page source it will not be present there.

Secondly, Even though you can get the headlines from the h2 tag, some other h2 tags are also present in the page. So, we need to isolate the headlines by using the class name of a parent div and then get the output.

import requests
from bs4 import BeautifulSoup
base_url = 'https://www.nytimes.com/'
r = requests.get(base_url)
soup = BeautifulSoup(r.text, 'lxml')
headline_text = soup.find('div',class_="css-11bbiel").find_all('h2')
for headline in headline_text:
    print(headline.text)

Output

Brexit Deal Fails in Parliament; May Faces No-Confidence Vote
Brexit, explained: Here’s what it all means.
Here’s what could happen next.
William Barr Vows to Protect Justice Dept. Integrity
Court Blocks Trump Administration From Asking About Citizenship in Census
Here are highlights from the Senate confirmation hearing.
House Votes to Condemn White Supremacy After King Comments
King Loses Committee Seats Over Remark
We put together a timeline of Mr. King’s history of racist actions.
Democrats Jilt Trump on Lunch but Look for Shutdown Exit
‘The Shutdown Makes Me Nervous’: Young People Caught in Impasse
Shutdown turmoil at a New York jail: Prisoners went on a hunger strike after family visits were canceled over staffing shortages.
Ex-Mexican President Took $100 Million Bribe, El Chapo Trial Witness Says
Last week, The Times reported on how a Colombian I.T. expert helped the authorities take down the kingpin.
Carol Channing, Larger-Than-Life Broadway Star, Dies at 97
Even From Afar, Channing Served Up That Broadway Wow
Theater colleagues recalled Ms. Channing as a tireless performer and promoter who had little use for doctors’ orders.
Britain Is a Nation in Desperate Need of a Driver
Why Steve King’s Punishment Took So Long 
Next to a National Park, People Plan for Winter. No One Planned for This.
How to Make New York as Progressive on Criminal Justice as Texas
The Cruelty of Call-Out Culture
Donald Trump and His Team of Morons
Republicans Condemn Steve King’s Racism? How Convenient
Our National Emergency Turns 2
Is 2019 Over Yet?
Donald Trump: The Russia File
Actually, the Numbers Show That We Need More Immigration, Not Less
Writer Moves From ‘Moonlight’ to Broadway, and Beyond
The Gay Penguins of Australia
Benno, Proudly Out of Step With the Age

Also, this time we were fortunate that the page just used javascript only for altering the style. This need not always be the case. Alternatively, you can use selenium.

user_108255560

Well, I really don't see the class name you mentioned. If you see the page source, all the headlines are in tag "h2". Try the below code and you can extract the text further from the output.

import requests
from bs4 import BeautifulSoup

base_url = 'https://www.nytimes.com/'
r = requests.get(base_url)
soup = BeautifulSoup(r.text, 'lxml')

headline_text = soup.find_all('h2')

for headline in headline_text:
    print(headline)
user_26791710
Related