Web scraper using BeautifulSoup not working

user_14636954 0
Viewed 138

I have coded a web scraper for Stack Overflow, but it doesn't work. Apparently, in my soup there are NoneType objects that just came from nowhere. Here's the web scraper code:

import requests
from bs4 import BeautifulSoup

url = 'https://stackoverflow.com/questions?tab=newest&page='

r = requests.post(url)
soup = BeautifulSoup(r.text, 'lxml').find('div', id='questions').find_all('div')

for summary in soup: # FIXME: Prints each question twice
    try:
        print(f'Question: {summary.h3.text}')
        print(f'Tags: {", ".join(summary.find("div", class_="tags").text[1:].split(" "))}')
    except Exception as e:
        print(e) # Prints "'NoneType' has no attribute 'text'" which shouldn't be in the soup

The error I get (If you didn't read the comment) is "'NoneType' has no attribute 'text'" which confuses me a lot because of the fact that there are NoneType objects in the soup.

I am using:

  • Windows 10
  • Python 3.8
python-requests web-scraping beautifulsoup python-3.x python
2 Answers

There aren't None type objects in soup, rather the calls to

... summary.h3 ...
... summary.find("div", class_="tags") ...

Within your loop will return None in the case that summary does not have a h3 element, or if there are no div elements with a class attribute of tags in the current summary respectively.

An Exception is naturally thrown in these cases as you attempt to access the text attribute of a None type element.

It could be helpful to use enumerate to show the indices of soup which are causing this Exception to be thrown.

e.g.

for i, summary in enumerate(soup):
    try:
        print(f'Question: {summary.h3.text}')
        print(f'Tags: {", ".join(summary.find("div", class_="tags").text[1:].split(" "))}')
    except Exception as e:
        print(f'Exception raised during handling of soup[{i}]')
        print(e)

It is probably advisable to use the above to gain a deeper understanding of the response structure so that you might best ignore these elements during parsing.

Of course you could always do something lazy like the below to pass over any AttributeErrors thrown within the loop but still print any other kind of Exception thrown - though this is obviously not the most efficient approach.

for summary in soup:
    try:
        print(f'Question: {summary.h3.text}')
        print(f'Tags: {", ".join(summary.find("div", class_="tags").text[1:].split(" "))}')
    except AttributeError as ee:
        pass
    except Exception as e:
        print(e)
user_120459070

JP193 has given you the answer in regard to why you get a NoneType returned.

The big problem that I have found with you code is that each of the <div> you find using BeautifulSoup(r.text,xml').find('div', id='questions').find_all('div') arent needed to get the "question" and the "tags", that is why you get a lot of AttributeError messages. To condense the <div> that are relevant you need to look closer at the code you get as a result of soup. To best condense the outcome of soup would be to find all the <div> with the class being "question-summary". So the code should look more like:

soup = BeautifulSoup(r.content, "html.parser").find("div", id="questions").find_all("div", {"class": "question-summary"})

After that point it is up to you how you print the results for the 'question' and 'tags'. I have done it like this:

    try:
        question = q.find("div", {"class": "summary"}).find('h3')
        print('Question: ' + question.text)

        tags = soup[0].find("div", {"class": "tags"}).text.split()
        print('Tags: ' + ', '.join(tags))
    except AttributeError:
        print('AttributeError div ' + str(int + 1))

Put together my solutions is:

from bs4 import BeautifulSoup

url = 'https://stackoverflow.com/questions?tab=newest&page='

r = requests.get(url)
soup = BeautifulSoup(r.content, "html.parser").find("div", id="questions").find_all("div", {"class": "question-summary"})

for int, q in enumerate(soup):
    try:
        question = q.find("div", {"class": "summary"}).find('h3')
        print('Question: ' + question.text)

        tags = soup[0].find("div", {"class": "tags"}).text.split()
        print('Tags: ' + ', '.join(tags))
    except AttributeError:
        print('AttributeError div ' + str(int + 1))
user_138064070
Related