I'm trying to get all the table in this url = "https://www.topuniversities.com/university-rankings/university-subject-rankings/2021/psychology".
The problem is that there's no table tag and neither <tr> and <td> tags. All the data in rows are in nested "div" tags.
The code I'm using is this:
from bs4 import BeautifulSoup
from selenium import webdriver
from webdriver_manager.firefox import GeckoDriverManager
import time
driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
driver.maximize_window()
driver.get(url)
time.sleep(5)
content = driver.page_source.encode('utf-8').strip()
soup = BeautifulSoup(content,"html.parser")
driver.quit()
print(soup)
Also, I'm only getting data from one column (column named "Overall Score") in the nested <div> tags.
Something else I realised is that there's only data from the 10 first rows in the soup output, but I'm trying to get all the 302 rows data.
Thanks a lot for any advise you coud give me.
EDIT
I managed to get what I expected following @KunduK's answer. This is the code I used at the end:
res = requests.get('https://www.topuniversities.com/sites/default/files/qs-rankings-data/en/3519089_indicators.txt?1614801117').json()
df = pd.DataFrame(res["data"])
df = df[["uni", "region", "location", "city", "overall",
"ind_69", "ind_70", "ind_76", "ind_77"]]
headers = {"uni":"University", "overall": "Overall Score", "ind_69": "H-index Citations",
"ind_70": "Citations per Paper", "ind_76": "Academic Reputation", "ind_77": "Employer Reputation"}
df.rename(columns=headers, inplace=True)
for column in headers.values():
df[column] = df[column].apply(lambda value: BeautifulSoup(value, 'html.parser').find('div').text)
df
The DataFrame is the following:
