Beautiful Soup Query with Filtering in Python

user_13356730 0
Viewed 446

I am trying to save the contents of each article in its own text file. What I am having trouble with is coming up with a beautiful soup approach that returns articles of the type News only while ignoring the other article types.

Website in question: https://www.nature.com/nature/articles

Info

  • Every article is enclosed in a pair of <article> tags
  • Each article type is hidden inside a <span> tag containing the data-test attribute with the article.type value.
  • Title to the article is placed inside the <a> tag with the data-track-label="link" attribute.
  • The article body wrapped in the <div> tag (look for "body" in the class attribute).

Current code

I was able to get up to the point where I can query the <span> for articles of the News type, but am struggling to take the next steps to return the other article specific information.

How can I take this further? For the articles of the the type News, I'd like to also be able to return that article's title and body while ignoring the other articles that are not of type News?

# Send HTTP requests
import requests

from bs4 import BeautifulSoup


class WebScraper:

    @staticmethod
    def get_the_source():
        # Obtain the URL
        url = 'https://www.nature.com/nature/articles'

        # Get the webpage
        r = requests.get(url)

        # Check response object's status code
        if r:
            the_source = open("source.html", "wb")

            soup = BeautifulSoup(r.content, 'html.parser')

            type_news = soup.find_all("span", string='News')

            for i in type_news:
                print(i.text)

            the_source.write(r.content)

            the_source.close()

            print('\nContent saved.')
        else:
            print(f'The URL returned {r.status_code}!')


WebScraper.get_the_source()

Sample HTML for an article that is of type News

The source code has 19 other articles with similar and different article types.

   <article class="u-full-height c-card c-card--flush" itemscope itemtype="http://schema.org/ScholarlyArticle">
        
            
                
                    <div class="c-card__image">
                        <picture>
                            <source
                                type="image/webp"
                                srcset="
                                    //media.springernature.com/w165h90/magazine-assets/d41586-021-00485-2/d41586-021-00485-2_18927840.jpg?as=webp 160w,
                                    //media.springernature.com/w290h158/magazine-assets/d41586-021-00485-2/d41586-021-00485-2_18927840.jpg?as=webp 290w"
                                sizes="
                                    (max-width: 640px) 160px,
                                    (max-width: 1200px) 290px,
                                    290px">
                            <img src="//media.springernature.com/w290h158/magazine-assets/d41586-021-00485-2/d41586-021-00485-2_18927840.jpg"
                                 alt=""
                                 itemprop="image">
                        </picture>
                    </div>
                
            
        
        <div class="c-card__body u-display-flex u-flex-direction-column">
            <h3 class="c-card__title" itemprop="name headline">
                <a href="/articles/d41586-021-00485-2"
                   class="c-card__link u-link-inherit"
                   itemprop="url"
                   data-track="click"
                   data-track-action="view article"
                   
                   data-track-label="link">Mars arrivals and Etna eruption — February's best science images</a>
            </h3>
            
                
                    <div class="c-card__summary u-mb-16 u-hide-sm-max"
                         itemprop="description">
                        <p>The month’s sharpest science shots, selected by <i>Nature's</i> photo team.</p>
                    </div>
                
            
            <div class="u-mt-auto">
                
                    <ul data-test="author-list" class="c-author-list c-author-list--compact u-mb-4">
                        <li itemprop="creator" itemscope="" itemtype="http://schema.org/Person"><span itemprop="name">Emma Stoye</span></li>
                    </ul>
                
                <div class="c-card__section c-meta">
                    <span class="c-meta__item c-meta__item--block-at-xl" data-test="article.type">
                        <span class="c-meta__type">News</span>
                    </span>
                    
                    
                        <time class="c-meta__item c-meta__item--block-at-xl" datetime="2021-03-05" itemprop="datePublished">05 Mar 2021</time>
                    
                </div>
            </div>
        </div>
    </article>


    
</div>
                    </li>
                
                    <li class="app-article-list-row__item">
                        <div class="u-full-height" data-native-ad-placement="false">
web-scraping beautifulsoup python
2 Answers

The simplest way, and you get more results per hit, is to add News into the query string as a param

https://www.nature.com/nature/articles?type=news

import requests
from bs4 import BeautifulSoup as bs

r = requests.get('https://www.nature.com/nature/articles?type=news')
soup = bs(r.content, 'lxml')
news_articles = soup.select('.app-article-list-row__item')

for n in news_articles:
    print(n.select_one('.c-card__link').text)

A variety of params for page 2 of news:

https://www.nature.com/nature/articles?searchType=journalSearch&sort=PubDate&type=news&page=2

If you monitor the browser network tab whilst manually filtering on the page, or selecting different pages numbers, you can see the logic of how the querystrings are constructed and tailor your requests accordingly e.g.

https://www.nature.com/nature/articles?type=news&year=2021

Otherwise, you could do more convoluted (in/ex)clusion with css selectors, based on whether based on whether article nodes have a specific child containing "News" (inclusion); exclusion beings News with another word/symbol (as per categories list):

import requests
from bs4 import BeautifulSoup as bs

r = requests.get('https://www.nature.com/nature/articles')
soup = bs(r.content, 'lxml')
news_articles = soup.select('.app-article-list-row__item:has(.c-meta__type:contains("News"):not( \
                            :contains("&"), \
                            :contains("in"), \
                            :contains("Career"), \
                            :contains("Feature")))') #exclusion n

for n in news_articles:
    print(n.select_one('.c-card__link').text)

You can remove categories from the :not() list if you want News & or News In etc...

user_62412350

if you don't want to filter the URL, loop through <article> then check element text for class c-meta__type

articles = soup.select('article')
for article in articles:
    article_type = article.select_one('.c-meta__type').text.strip()
    if article_type == 'News':
    # or if type contain News
    # if 'News' in article_type:
        title = article.select_one('a').text
        summary = article.select_one('.c-card__summary p').text
        print("{}: {}\n{}\n\n".format(article_type, title, summary))
user_40823440
Related