Website scraping with Python, how do I know where to reference in the html?

user_15339219 0
Viewed 42

I'm a complete beginner who has only built basic Python projects. Right now I'm building a scraper in Python with bs4 to help me read success stories off of a website. These success stories are all in a table, so I thought I would find an html tag that said table and would encompass the entire table.

However, it is all just <div and <span class, and when I use soup.find("div") or ("span") it returns only the single word "div" or "span". This is what I have so far, and I know it isn't right or set up correctly but I'm too inexperienced to know why yet.

from bs4 import BeautifulSoup
from urllib.request import Request, urlopen
import requests
req = Request('https://www.calix.com/about-calix/success-stories.html', headers={'User-Agent': 'Mozilla/5.0'})
webpage = urlopen(req).read()
soup = BeautifulSoup(webpage, "lxml")
soup.find("div", {"id": "content-calix-en-site-prod-home-about-calix-success-stories-jcr-content"})
print('div')

I have watched several tutorials on how to use bs4 and I have successfully scraped basic websites, but all I can do for this one is get ALL of the html, not the chunks I need (just the success stories).

beautifulsoup class html python
2 Answers

You are printing 'div' make sure to be printing soup as soup gets updated whenever you find something within it.

You should have a look at the bs4 documentation.

user_116030430 
soup.find("div", {"id": "content-calix-en-site-prod-home-about-calix-success-stories-jcr-content"})

Here you're calling soup.find() but you're not saving the results into a variable, so the results are lost.

print('div')

And here you're printing the literal string div. I don't think that's what you intended.

Try something like this:

div = soup.find("div", {"id": "..."})
print(div)
user_4941340
Related