Duplicated results while scraping the website

Question

I have the following script which scraps all the information from a website. However, when I run it, I get duplicated records of the blogs.

import requests
import pandas as pd
import numpy as np
from time import sleep
from random import randint
from bs4 import BeautifulSoup
from urllib.parse import urlparse
import re

blog_topics = []
page = "https://www.bartonassociates.com/blog/"
soup = BeautifulSoup(requests.get(page).content, 'html.parser')
for link in soup.find_all(href=re.compile("/blog/tag")):
    url = link.get('href')
    if '/blog/tag/p' not in urlparse(link.get('href')).path:
        blog_topics.append(url)
    else:
        pass
 
# VARIABLE TO DEFINE A RANGE BASED ON NO.OF PAGES
pages = np.arange(1)

# DEFINING CUSTOM VARIABLES
title_blognames_links_ = []
author_and_dates_ = []

# LOOP TO RETRIEVE TITLE, BLOG NAMES, LINKS, AUTHORS AND DATE PUBLISHED
for page in pages:
    for blogs in blog_topics:
        blog_url= blogs +'/p' + str(page) 
        sleep(randint(2,7))
        soup = BeautifulSoup(requests.get(blog_url).content, 'html.parser')
         
        #Information on title, blog names and their links
        for h4 in soup.select("h4"):
                for h2 in soup.select("h2"):
                    title_blognames_links_.append((h4.get_text(strip=True), h4.a["href"], h2.get_text(strip=True).replace('"',"")[11:]))
                
        #Information of authors and dates
        for tag in soup.find_all(class_="author"):
                author_and_dates_.append(tag.get_text(strip=True))

I believe it has to do something with the pages = np.arange(1) range I have provided. P.S. (1) was just a trail. I have tried (1,17),(1),(2)

Background: The maximum pages in a blog topic I have is 17, each topic has 10 blogs each (approx)

What I am looking for, is to get all unique blog information from all the blog topics

Not sure what I am doing wrong here

user_10035985 · Answer

To get all information from all topics you can first grab all topic links (you've done that in your code too) and then for each topic get all pages and all information (not other way around):

import re
import requests
import pandas as pd
from bs4 import BeautifulSoup


url = "https://www.bartonassociates.com/blog"
soup = BeautifulSoup(requests.get(url).content, "html.parser")
topics = [
    a["href"] for a in soup.select('h3:-soup-contains("Blog Topics") + ul a')
]

all_data = []

for t in topics:
    while True:
        soup = BeautifulSoup(requests.get(t).content, "html.parser")

        topic_name = re.search(
            r'"([^"]+)"', soup.select_one("h2").get_text(strip=True)
        ).group(1)

        for entry in soup.select(".blog-entry"):
            title = entry.h4.get_text(strip=True)

            title = entry.h4.get_text(strip=True)
            link = entry.a["href"]

            tmp = entry.select_one(".author").get_text(strip=True)
            if tmp:
                author, date = map(
                    str.strip,
                    entry.select_one(".author").get_text(strip=True).split("|"),
                )
            else:
                author, date = "N/A", "N/A"

            all_data.append([topic_name, title, link, author, date])
            print(topic_name, title, link, author, date, sep="
")
            print()

        t = soup.select_one('a:-soup-contains("View More")')
        if not t:
            break
        t = t["href"]


df = pd.DataFrame(
    all_data, columns=["topic", "title", "link", "author", "date"]
)
print(df)
df.to_csv("data.csv", index=False)

Prints:

Healthcare News and Trends
DO vs. MD: What’s the Difference?
https://www.bartonassociates.com/blog/whats-the-difference-do-md
Tayla Holman September 09, 2021

Healthcare News and Trends
What is “The Great Resignation?”
https://www.bartonassociates.com/blog/what-is-the-great-resignation
Chris Keeley September 02, 2021

Healthcare News and Trends
CME Requirements for Physicians by State
https://www.bartonassociates.com/blog/cme-requirements-for-physicians-by-state
Teresa Otto, MD July 15, 2021

...and so on.

and saves data.csv (screenshot from LibreOffice):

Duplicated results while scraping the website

1 Answers