How to scrape words from title and body of web pages through URLs in a pandas data frame column and add them both to separate columns?

sonar 0
Viewed 37

Question title explains it all. Here's a sample data frame:

df_sample = pd.DataFrame({'page': {1225: 'https://nextgrowthlabs.com/en/know-about-the-app-screenshot-guidelines-of-the-apple-app-store/',
  1229: 'https://nextgrowthlabs.com/en/know-about-the-app-screenshot-guidelines-of-the-apple-app-store/',
  241: 'https://nextgrowthlabs.com/learn-everything-about-app-screenshot-size-guidelines/',
  1175: 'https://nextgrowthlabs.com/en/tracking-user-level-interactions-with-bigquery/',
  141: 'https://nextgrowthlabs.com/may-2022-google-play-update/'}})

df_sample

Now I want to scrape words from title and words from body of the web pages found through these URLs and add them both to separate columns in this same data frame. Can you at least point me in the right direction?

beautifulsoup web-scraping dataframe pandas python
1 Answers

I would use the requests package to load the page content, and some regex to parse the title for instance

import requests
import re

for url in df_sample["page"]:
    response = requests.get(url)
    title = re.search('<\W*title\W*(.*)</title', response.text, re.IGNORECASE).group(1)
    # process title here

    body = response.content
    # process content here

Then do what you want with your new processed data. (title regex has been taken from How to get page title in requests)

Hope this helps !

PlainRavioli0
Related