how to use beautifulSoup to extract html5 elements such as <section>

user_3983454 0
Viewed 299

I intend to extract the article text from an NYT article. However I don't know how to extract by html5 tags such as section name.

import urllib.request
from bs4 import BeautifulSoup

html = urllib.request.urlopen('https://www.nytimes.com/2019/10/24/opinion/chuck-schumer-electric-car.html?action=click&module=Opinion&pgtype=Homepage')
soup = BeautifulSoup(html)
data = soup.findAll(text=True)

The main text is wrapped in a section named 'articleBody'. What kind of soup.find() syntax can I use to extract that? enter image description here

web-scraping beautifulsoup html python
2 Answers

The find method searches tags, it doesn't differentiate HTML5 from any other (X)HTML tag name 

article = soup.find("section",{"name":"articleBody"})
user_23086830

You can scrape the pre-loaded data from script tag and parse with json library. The first code block brings back a little more content than you wanted.

You can further restrict by looking up ids of paragraphs within body, and use those to filter content, as shown in bottom block; You then get exactly the article content you describe.

import requests, re, json

r = requests.get('https://www.nytimes.com/2019/10/24/opinion/chuck-schumer-electric-car.html?action=click&module=Opinion&pgtype=Homepage')
p = re.compile(r'window\.__preloadedData = (.*})')
data = json.loads(p.findall(r.text)[0])

for k,v in data['initialState'].items():
    if k.startswith('$Article') and 'formats' in v:
        print(v['text@stripHtml'] if 'text@stripHtml' in v else v['text'])

You can explore the json here: https://jsoneditoronline.org/?id=f9ae1fb774af439d8e9b32247db9d853

The following shows how to use additional logic to limit to just output you want:

ids = []

for k,v in data['initialState'].items():
    if k.startswith('$Article') and v['__typename'] == 'ParagraphBlock' and 'content' in v:
        ids += [v['content'][0]['id']]

for k,v in data['initialState'].items():
    if k in ids:
        print(v['text'])
user_62412350
Related