I keep getting an empty result when trying to webscrape

David Henson 0
Viewed 29
import requests
from bs4 import BeautifulSoup
response = requests.get("https://www.stackoverflow.com/questions")

soup = BeautifulSoup(response.text, "html.parser")
questions = soup.select(".question-summary")
print(questions)

with this code I keep getting the result - []

If I add a [0] to the print(questions[0]), I get the error - IndexError: list index out of range

Any ideas? I've tried really hard to try and solve this my self but nothing seems to be working for me. I'm new to python so maybe missing some modules or something?

python
1 Answers

There are no elements in the response that I could find that actually have the class question-summary, so it's not surprising you get no results.

There are elements like these:

<div id="question-summary-73709434" class="s-post-summary js-post-summary" data-post-id="73709434" data-post-type-id="1">

Which you could get with:

soup.select('.s-post-summary')

But you were likely after the text of the summary, something like:

import requests
from bs4 import BeautifulSoup

response = requests.get("https://www.stackoverflow.com/questions")

soup = BeautifulSoup(response.text, "html.parser")
for element in soup.select('.s-post-summary--content-excerpt'):
    print(element.text.strip())
Grismar0
Related