How to get simple information through a crawler

Question

I am trying to make a simple crawler that scrapes through this https://en.wikipedia.org/wiki/Web_scraping page, then proceeds to extract the 19 links from the See About section. This I manage to do, however I am also trying to extract the first paragraph from each of those 19 links and this is where it stops "working". I get the same paragraph from the first page and not from each one. This is what I have so far. I know there might be better options for doing this but i want to stick to BeautifulSoup and simple python code.

from urllib.parse import urljoin
import requests
from bs4 import BeautifulSoup

url = 'https://en.wikipedia.org/wiki/Web_scraping'

data = requests.get('https://en.wikipedia.org/wiki/Web_scraping').text

soup = BeautifulSoup(data, 'html.parser')


def visit():
    try:
        p = soup.p
        print(p.get_text())
    except AttributeError:
        print(' Tag was not found')


links_todo = []
links = soup.find('div', {'class': 'div-col'}).find_all('a')
for link in links:
    if 'href' in link.attrs:
        links_todo.append(urljoin(url, link.attrs['href']))

while links_todo:
    url_to_visit = links_todo.pop()
    print('Now visiting:', url_to_visit)
    visit()

Example of the first print

Now visiting: https://en.wikipedia.org/wiki/OpenSocial
Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites. The web scraping software may directly access the World Wide Web using the Hypertext Transfer Protocol or a web browser. While web scraping can be done manually by a software user, the term typically refers to automated processes implemented using a bot or web crawler. It is a form of copying in which specific data is gathered and copied from the web, typically into a central local database or spreadsheet, for later retrieval or analysis.

Intended function should be that it prints the first paragraph for every new link printed, not the same paragraph from the first link. What do I need to do in order to fix this? Or any tips on what I am missing. I am fairly new to python so I am still learning the concepts as I work on things.

user_2817133 · Answer

At the top of your code you define data and soup. Both are tied to https://en.wikipedia.org/wiki/Web_scraping.

Every time you call visit(), you print from soup, and soup never changes.

You need to pass the url to visit(), e.g. visit(url_to_visit). The visit function should accept the url as an argument, then visit the page using requests, and create a new soup from the returned data, then print the first paragraph.

Edited to add code explaining my original answer:

from urllib.parse import urljoin
import requests
from bs4 import BeautifulSoup
 
start_url = 'https://en.wikipedia.org/wiki/Web_scraping'
# Renamed this to start_url to make it clear that this is the source page 
data = requests.get(start_url).text
 
soup = BeautifulSoup(data, 'html.parser')
 
 
def visit(new_url): # function now accepts a url as an argument
    try:
        new_data = requests.get(new_url).text # retrieve the text from the url
        new_soup = BeautifulSoup(new_data, 'html.parser') # process the retrieved html in beautiful soup
        p = new_soup.p
        print(p.get_text())
    except AttributeError:
        print(' Tag was not found')
 
 
links_todo = []
links = soup.find('div', {'class': 'div-col'}).find_all('a')
for link in links:
    if 'href' in link.attrs:
        links_todo.append(urljoin(start_url, link.attrs['href']))
 
while links_todo:
    url_to_visit = links_todo.pop()
    print('Now visiting:', url_to_visit)
    visit(url_to_visit) # here's where we pass each line to the visit() function

How to get simple information through a crawler

1 Answers