Could not find a class name for BeautifulSoup on the website to crawl on

user_9340081 0
Viewed 53

I am new to Beautiful Soup.

I am trying to get the "Ranking Criteria" class in the below link. Unfortunately, I used the "criteria" as its class for soup_findAll(), but it showed no content there. I could not find any other class names which can give me the data I want (Overall score, academic reputation and so on)

I actually wanted to do web crawling for multiple universities, so I hope to use the URL which I can format for various universities(Just change the universities name). Otherwise, I would have just used the outerHTML for that(tested and it worked, but I did not know how to customize that for multiple universities)

My code is as below. I ended up using get_text():

r = requests.get("https://www.topuniversities.com/universities/california-institute-technology-caltech")
html = r.text
soup = BeautifulSoup(html, 'html.parser')
tds = soup.get_text()

print(tds)

It was not successful, as it got too much stuff which left me hard to evaluate the information I want.

Any help would be highly appreciated! Thanks!

The link I am trying to scrape:

beautifulsoup python-3.x
1 Answers

The data is loaded dynamically via JavaScript Ajax request. But you could use requests module to simulate it.

For example:

import requests
from bs4 import BeautifulSoup

url = 'https://www.topuniversities.com/universities/california-institute-technology-caltech'

soup = BeautifulSoup( requests.get(url).content, 'html.parser' )
ajax_url = 'https://www.topuniversities.com' + soup.select_one('a.use-ajax')['href'].replace('nojs', 'ajax')
data = requests.post(ajax_url).json()

for d in data:
    if 'data' in d:
        soup = BeautifulSoup(d['data'], 'html.parser')
        break

for div in soup.select('div.criteria'):
    criteria = div.find(text=True).strip()
    ranking = div.b.get_text(strip=True)
    print('{:<30} {}'.format(criteria, ranking))

Prints:

Overall Score:                 97
Academic Reputation:           97
Employer Reputation:           82.8
Faculty Student:               100
Citations per Faculty:         99.9
International Faculty:         100
International Students:        88.2
user_100359850
Related