I am trying to make a web scraper that checks for all the courses in a coursera specialisation..and then gives the list of them

Admin 1
Viewed 114

The problem is there is a button saying "SHOW MORE" on the webpage...because of that the script can't access all the courses .... from bs4 import BeautifulSoup import requests

res = requests.get('https://www.coursera.org/specializations/digital-manufacturing-design- 
technology#courses')
txt = res.text
status = res.status_code
#print(txt)
#print(status)## 200 is the code for success

page =  requests.get('https://www.coursera.org/specializations/digital-manufacturing-design- 
technology#courses')
soup = BeautifulSoup(page.content,'lxml')

#Display the title of the specialisation
specialization_title = soup.find('h1')
print(specialization_title.text)
print("\n")

#Display the courses inside the specialisation
number_of_courses = soup.find('h2',class_ = 'headline-4-text bold m-b-1')
print(number_of_courses.text)
print("\n")

course_cards = soup.find_all('h3',class_= 'headline-3-text bold m-t-1 m-b-2')
for course in course_cards:
    print(course.text)
web web-scraping beautifulsoup
1 Answers

The data is stored inside the page in form of javascript object. You can use re/json module to decode it:

import re
import json
import requests
from textwrap import shorten


url = "https://www.coursera.org/specializations/digital-manufacturing-design-technology#courses"
html_doc = requests.get(url).text
data = json.loads(
    re.search(r"window.__APOLLO_STATE__ = (.*});", html_doc).group(1)
)

# uncomment to print all data:
# print(json.dumps(data, indent=4))

i = 1
for k, v in data.items():
    if "SDPCourse:" in k and "." not in k:
        print(
            "{:<3} {:<8} {:<60} {}".format(
                i,
                v["averageInstructorRating"],
                v["name"],
                shorten(v["description"], 40),
            )
        )
        i += 1

Prints:

1   4.6      Digital Manufacturing & Design                               This course will expose you to the [...]
2   4.56     Digital Thread: Components                                   This course will help you [...]
3   4.76     Digital Thread: Implementation                               There are opportunities throughout [...]
4   4.41     Advanced Manufacturing Process Analysis                      Variability is a fact of life in [...]
5   4.56     Intelligent Machining                                        Manufacturers are increasingly [...]
6   4.55     Advanced Manufacturing Enterprise                            Enterprises that seek to become [...]
7   4.55     Cyber Security in Manufacturing                              The nature of digital [...]
8   4.54     MBSE: Model-Based Systems Engineering                        This Model-Based Systems [...]
9   4.68     Roadmap to Success in Digital Manufacturing & Design         Learners will create a roadmap to [...]
user_100359850
Related