Removing unwanted tags in Python using BeautifulSoup

user_13794360 0
Viewed 49

I am trying to create a simple web scraper that will return the list of names from this website: https://www.verywellfamily.com/top-1000-baby-girl-names-2757832 However, I cannot figure out how to get them in a 'clean' format. This code below returns the names, but all the tags are still with it.

import requests
import urllib.request
from bs4 import BeautifulSoup

URL = 'https://www.verywellfamily.com/top-1000-baby-girl-names-2757832'
page = requests.get(URL)
soup = BeautifulSoup(page.content, 'html.parser')

names = soup.find(id='mntl-sc-block_1-0-13')

print(names)

This code is to put it into a file:

file_name = URL.rsplit('/',1)[1].rsplit('.')[0]
with open('./{}.txt'.format(file_name), mode='wt', encoding='utf-8') as file:
    file.write((names))
web-scraping beautifulsoup python
1 Answers

You can use the separator='\n' argument in the get_text() method:

import requests
from bs4 import BeautifulSoup

URL = "https://www.verywellfamily.com/top-1000-baby-girl-names-2757832"
soup = BeautifulSoup(requests.get(URL).content, "html.parser")

for tag in soup.find_all(id='mntl-sc-block_1-0-13'):
    print(tag.get_text(separator='\n'))

Edit for printing with a counter (like the website):

for count, tag in enumerate(
    soup.find(id="mntl-sc-block_1-0-13").get_text(separator="\n").split("\n")[2:],
    start=1,
):
    if not tag:
        continue
    print(count, tag)

Output:

1 Olivia
2 Emma
3 Ava
4 Sophia
5 Isabella
6 Charlotte
...
998 Zendaya
999 Ariadne
1000 Dixie

Edit 2 writing to a file:

file_name = URL.split(".com/")[1]

with open("./{}.txt".format(file_name), "wt", encoding="utf-8") as f:
    for count, tag in enumerate(
        soup.find(id="mntl-sc-block_1-0-13").get_text(separator="\n").split("\n")[2:],
        start=1,
    ):
        if not tag:
            continue
        f.write("{0} {1} \n".format(count, tag))
user_123497340
Related