Trouble scraping data with Beautifulsoup

user_17818154 0
Viewed 39

I am fairly new to using python. I want to create a web-scrpaing tool which allows me to scrape through 90 sites which are pre-collated in the form of an excel spreadsheet.

An example URL of what I'm trying to scrape is below: https://mycare.derby.gov.uk/Services/326

I have watched a few tutorials on how to use Beautifulsoup to achieve this and although these have been largely helpful in scraping some data, I am struggling to scrape the rest for two reasons:

  1. Some of the data is in list form, specifically, I am trying to scrape 'CQC Service Type' and 'CQC Specialism' data but I can't work out how to code for this.

  2. Some of the data is embedded into the website using a javascript window - specifically I want to scrape data on 'CQC overall rating' and 'Date of inspection'.

Here is my code so far:

import pandas as pd  
import requests  
from bs4 import BeautifulSoup 
import http.client

http.client._MAXHEADERS = 1000


file="/Users/dhesis/PycharmProjects/pythonProject/CH_ID (2).xlsx"
Carehome_list = pd.read_excel(file).iloc[:, 0].to_list()
print(Carehome_list)

def loop_excel_sheet ():
    for CH in Carehome_list[:2]:
        url = CH
        headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36'}
        r = requests.get(url, headers=headers)
        soup = BeautifulSoup(r.content, 'html.parser')
        divs = soup.find_all('div', class_ ="container-fluid bem-js-service-details")
        div_CQC = soup.find_all('dl', class_="ai-group-fields read-data bem-description-list bem-description-list--ai-group")
        div_cqc_Rating = soup.find_all('span', class_="cqc-widget-overall requires-improvement" )
        for item in divs:
            name = item.find('dd', itemprop = 'branchOf').text
            address = item.find('span', itemprop = 'location').text
            try:
                website = item.find('a', itemprop = 'url').text
            except:
                website = ''
            last_updated = item.find('p').text.strip()
            try:
                telephone = item.find('dd', itemprop='telephone').text
            except:
                telephone = ''

            info = {
                'Name': name,
                'Address': address,
                'Website': website,
                'Telephone': telephone,
                'Website Last Updated': last_updated,
            }
            chlist.append(info)


chlist = []
loop_excel_sheet()
df= pd.DataFrame(chlist)
df.to_csv('CH Tester.csv')


#Remaining sections to be scraped
#CQC Specialism
#CQC service type
#CQC overall rating
#Date of CQC rating

Hopefully someone can help.

web-scraping beautifulsoup python-3.x
1 Answers

As for number 2:

I did a bit of reverse engineering on the website, and it seems that the page contains a <script> tag, the src for which contains some internal ID for the location. This script then makes a request to a URL that returns some JS code that then runs a callback and embeds the widget into the page. In order to extract this information, you will need to perform these requests manually, and then extract the relevant information from the response. Here's a proof of concept code that does this and that you can adapt to your needs:

import requests
from bs4 import BeautifulSoup
from urllib.parse import urlparse, parse_qs
import json

SITE_URL = "https://mycare.derby.gov.uk/Services/316"
HEADERS = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36'}
CALLBACK_URL_TEMPLATE = "https://www.cqc.org.uk/widget/{}/CQCWidget-{}-1/location?callback=CQCWidgetDisplayWidget"
CALLBACK_START_PREFIX = """CQCWidgetDisplayWidget("""

url_resp = requests.get(SITE_URL, headers=HEADERS)
#Fail if 404 or other HTTP error
url_resp.raise_for_status()
url_text = url_resp.text
#I prefer to use html5lib, it's slow
#but way more lenient when dealing with 
#malformed HTML
url_soup = BeautifulSoup(url_text, "html5lib")

all_scripts = url_soup.find_all("script")
#Find the script with the code
widget_scripts = [s for s in all_scripts if "cqc_widget/widget.js" in s.attrs.get("src", "")]
#There should be only one such script on the page
assert len(widget_scripts) == 1
widget_script = widget_scripts[0]
widget_script_src = widget_script["src"]

#Extract the internal ID from the URL
url_parse = urlparse(widget_script_src)
location_internal_id = parse_qs(url_parse.query)["data-id"][0]

#Create the callback URL with the internal ID
callback_url = CALLBACK_URL_TEMPLATE.format(location_internal_id, location_internal_id)
#Get data from that URL
cb_url_resp = requests.get(callback_url, headers=HEADERS)
cb_url_resp.raise_for_status()
cb_url_text = cb_url_resp.text
#Here, we will need to do some more clever parsing
#First, we will need to remove the JS callback call
cb_url_json = cb_url_text.removeprefix(CALLBACK_START_PREFIX)
cb_url_json = cb_url_json.removesuffix(")")
#Next we will parse the rest of the response as JSON
cb_url_data = json.loads(cb_url_json)
#And verity that we are getting the right data
assert cb_url_data["id"] == f"CQCWidget-{location_internal_id}-1"
#We will then extract the widget HTML and parse into a BS object
cb_widget_html = cb_url_data["response"]
cb_widget_soup = BeautifulSoup(cb_widget_html, "html5lib")

#This is some vey simple (and probably unreliable) 
#code that extracts the necessary data from the object
status_elem = cb_widget_soup.find("span")
date_elem = status_elem.find_next()
status = status_elem.text
date = date_elem.text
#Print to screen
print(f"Status: {status}, inspection date: {date}")

For the URL in your question, it prints: Status: Requires improvement, inspection date: 31 May 2022. I've also tried it with other URL's from this website, and it appears to be working.

user_146290390
Related