Python Beautiful Soup find_all

Question

Hi I'm trying to get some information from a website. Pardon me if I formatted anything wrong this is my first time posting to SO.

soup.find('div', {"class":"stars"})

From this I receive

I need that "4.0 star rating"

When I use:

soup.find('div', {"class":"stars"})["title"]

It works, but not for find_all. But I'm trying to find all cases and put them into a list.

This is my full code below.

    def get_info():
        from IPython.display import HTML
        import requests
        from bs4 import BeautifulSoup
        n = 1
        for page in range(53):
            url = f"https://www.sitejabber.com/reviews/apple.com?page= 
   {n}&sort=Reviews.processed&direction=DESC#reviews"
            r = requests.get(url)
            soup = BeautifulSoup(r.text, 'lxml')
            all_reviews = soup.find_all('div', {'class':"truncate_review"})
            all_dates = soup.find_all('div', {'class':'review__date'},'title')
            all_titles = soup.find_all('span', {'class':'review__title__text'})
            reviews_class = soup.find('div', {"class":"review__stars"})
            for review in all_reviews:

    all_reviews_list.append(review.text.replace("
","").replace("	",""))
            for date in all_dates:

all_dates_list.append(date.text.replace("
","").replace("	",""))
            for title in all_titles:

  all_titles_list.append(title.text.replace("
","").replace("	",""))
            for stars in reviews_class.find_all('div', {'class':'stars'}):
                all_star_ratings.append(stars['title'])



            n += 1

Sorry my indenting got a bit messed up but that is my full code.

user_13062813 · Answer

Traverse through the bs4 element as you do in dictionary.
If you are using find():

soup.find('div', {"class":"stars"}) ['title']

this works since find() returns a single value.
But if you are using find_all(), it returns a list and list[string] is an invalid process.
Therefore, you can create a list of those:

res = []
for i in soup.find_all('div', {"class":"stars"}):
    res.append(i['title'])

else, as a one-liner:

res = [i['title'] for i in soup.find_all('div', {"class":"stars"})]

Since you want all titles of the reviews, you need to specify the review container, that is, scrape from:

So the code will be:

review = soup.find_all('div',class_="review__container")
res = [i['title'] for j in review for i in j.find_all('div',class_='stars')]

gives:

['1.0 star rating', '1.0 star rating', '3.0 star rating', '1.0 star rating', '1.0 star rating', '1.0 star rating', '1.0 star rating', '1.0 star rating', '1.0 star rating', '1.0 star rating', '5.0 star rating', '1.0 star rating', '1.0 star rating', '1.0 star rating', '1.0 star rating', '1.0 star rating', '1.0 star rating', '5.0 star rating', '2.0 star rating', '5.0 star rating', '1.0 star rating', '2.0 star rating', '1.0 star rating', '5.0 star rating', '1.0 star rating', '5.0 star rating']

user_4539709 · Answer

How about the following

from bs4 import BeautifulSoup

html = """
 
"""

soup = BeautifulSoup(html, features="lxml")
element = soup.select('.stars')[0]['title']
print(element)

Prints

4.0 star rating

using the url

import requests
from bs4 import BeautifulSoup

url = 'https://www.sitejabber.com/reviews/apple.com?page={n}&sort=Reviews.processed&direction=DESC#reviews'
page = requests.get(url=url)

soup = BeautifulSoup(page.text, features="lxml")

elements = soup.select('.stars')
# print(elements)

for element in elements:
    print(element['title'])

Prints

4.0 star rating
3.8 star rating
3.7 star rating
4.3 star rating
3.8 star rating
4.2 star rating
0.0 star rating
0.0 star rating
5.0 star rating
1.0 star rating
1.0 star rating
1.0 star rating
3.0 star rating
1.0 star rating
1.0 star rating
1.0 star rating
1.0 star rating
1.0 star rating
1.0 star rating
1.0 star rating
5.0 star rating
1.0 star rating
1.0 star rating
1.0 star rating
1.0 star rating
1.0 star rating
1.0 star rating
5.0 star rating
2.0 star rating
5.0 star rating
1.0 star rating
2.0 star rating
1.0 star rating
5.0 star rating
1.0 star rating
5.0 star rating
4.3 star rating
3.5 star rating
4.7 star rating
3.7 star rating
4.8 star rating
3.7 star rating

Python Beautiful Soup find_all

2 Answers