JSONDecodeError webscraping with Python3.9 and BeautifulSoup 4

Viewed 394

I'm attempting to scrape some TrustPilot reviews of a given brand - this is my code:

import requests
from bs4 import BeautifulSoup
import time
import json

headers = {"User-Agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0"}
#def get_total_items(url):

#soup = BeautifulSoup(requests.get(url, format(0),headers).text, 'lxml')
stars = []
dates = []
results = []
with requests.Session() as s:
    for num in range(1,2):
        url = "https://www.trustpilot.com/review/www.hiwaldo.com?page={}".format(num)
        r = s.get(url, headers = headers)
        soup = BeautifulSoup(r.content, 'lxml')

        for star in soup.find_all("section", {"class":"review__content"}):

            # Get rating value
            rating = star.find("div", {"class":"star-rating star-rating--medium"}).find('img').get('alt')

            # Get date value
            date_json = json.loads(star.find('script').text)
            date = date_json['publishedDate']

            stars.append(rating)
            dates.append(date)

            data = {"Rating": rating, "Date": date}
            results.append(data)

        time.sleep(2)


print(results)

When I run python3 ~/Desktop/reviews.py I encounter the following error messaging:

Traceback (most recent call last):
      File "/Users/user/Desktop/reviews.py", line 25, in <module>
        date_json = json.loads(star.find('script').text)
      File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 346, in loads
        return _default_decoder.decode(s)
      File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 337, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 355, in raw_decode
        raise JSONDecodeError("Expecting value", s, err.value) from None
    json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Is there anything glaringly obvious wrong with this set-up? I'm a complete python novice, in case that also wasn't obvious.

Many thanks in advance!

1 Answers

To extract the JSON data from star, use the .string method instead of .text.

So instead of:

date_json = json.loads(star.find('script').text)

Use:

date_json = json.loads(star.find('script').string)
Related