Do you guys know a way I could go the nth page (e.g. page 100) in this websites pagination without going through each and every page?
Here is the link to the website: https://www.sustainalytics.com/esg-ratings
(Note: Just an example, I am not collecting or selling this data)
I could also do it manually through chrome if there is a way.
Thank you
You would do something like this, with requests/BeautifulSoup:
import requests
from bs4 import BeautifulSoup
data = {
'industry': '',
'rating': '',
'filter': '',
'page': 100, ### this is where you would select the specific page
'pageSize': 100,
'resourcePackage': 'Sustainalytics'
}
r = requests.post('https://www.sustainalytics.com/sustapi/companyratings/getcompanyratings', data=data)
soup = BeautifulSoup(r.text, 'html.parser')
print(soup.get_text(strip=True))
This would return the text response of that actual table (which is not a table per se, you would need to further isolate and select elements from html response - see the elegant response from Andrej):
Sera Prognostics, Inc.NAS:SERA23.9Medium ESG RiskSerba Dinamik Holdings Bhd.KLS:527944.6Severe ESG RiskSerco Group PLCLON:SRP19.1Low ESG RiskSercomm Corp.TAI:538824.3Medium ESG RiskSeres Therapeutics IncNAS:MCRB35.9High ESG RiskSeria Co. Ltd.TKS:278221.7Medium [....]
And you would do something like this with Selenium (which in itself is not a tool designed to send POST requests, hence it will execute Javascript to get the data and replace the information in page):
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
chrome_options = Options()
chrome_options.add_argument("--no-sandbox")
webdriver_service = Service("chromedriver/chromedriver") ## path to where you saved chromedriver binary
browser = webdriver.Chrome(service=webdriver_service, options=chrome_options)
url='https://www.sustainalytics.com/esg-ratings'
browser.get(url)
print('sending xhr request..')
browser.execute_script('''
$.ajax({
type: 'POST',
url: 'https://www.sustainalytics.com/sustapi/companyratings/getcompanyratings',
data: 'industry=&rating=&filter=&page=100&pageSize=10&resourcePackage=Sustainalytics',
success: function(data){
$('#company_ratings').html(data);
}
});
''')
print('check page content, xhr response returned')
The table in page would be updated with page 100 information, and pagination would reflect that as well.
To get all data from pages into pandas dataframe you can use next example:
import requests
import pandas as pd
from bs4 import BeautifulSoup
url = "https://www.sustainalytics.com/sustapi/companyratings/getcompanyratings"
data = {
"industry": "",
"rating": "",
"filter": "",
"page": "1",
"pageSize": "10",
"resourcePackage": "Sustainalytics",
}
all_rows = []
for data["page"] in range(1, 3): # <-- increase the range here
soup = BeautifulSoup(requests.post(url, data=data).content, "html.parser")
for s in soup.select(".company-row"):
all_rows.append(s.get_text(strip=True, separator="\n").split("\n"))
all_rows[-1].append(
"https://www.sustainalytics.com/esg-rating" + s.a["data-href"]
)
df = pd.DataFrame(
all_rows, columns=["Name", "Symbol", "ESG Risk Rating", "Text", "Link"]
)
print(df)
Prints:
Name Symbol ESG Risk Rating Text Link
0 1-800-Flowers.com Inc NAS:FLWS 22.1 Medium ESG Risk https://www.sustainalytics.com/esg-rating/1-800-flowers-com-inc/1007902331
1 1&1 AG ETR:1U1 22.3 Medium ESG Risk https://www.sustainalytics.com/esg-rating/1-1-ag/1264207944
2 10X Genomics, Inc. NAS:TXG 22.6 Medium ESG Risk https://www.sustainalytics.com/esg-rating/10x-genomics-inc/2000259050
3 111, Inc. NAS:YI 28.7 Medium ESG Risk https://www.sustainalytics.com/esg-rating/111-inc/2006341104
4 17 Education & Technology Group, Inc. NAS:YQ 27.0 Medium ESG Risk https://www.sustainalytics.com/esg-rating/17-education-technology-group-inc/2008366023
5 1Life Healthcare, Inc. NAS:ONEM 24.6 Medium ESG Risk https://www.sustainalytics.com/esg-rating/1life-healthcare-inc/2000165824
6 1st Source Corp NAS:SRCE 31.7 High ESG Risk https://www.sustainalytics.com/esg-rating/1st-source-corp/1008054650
7 1stdibs.com, Inc. NAS:DIBS 28.0 Medium ESG Risk https://www.sustainalytics.com/esg-rating/1stdibs-com-inc/2001638549
8 22nd Century Group Inc NAS:XXII 31.7 High ESG Risk https://www.sustainalytics.com/esg-rating/22nd-century-group-inc/1032256453
9 2i Rete Gas SpA - 35.3 High ESG Risk https://www.sustainalytics.com/esg-rating/2i-rete-gas-spa/1008505575
10 2U, Inc. NAS:TWOU 19.8 Low ESG Risk https://www.sustainalytics.com/esg-rating/2u-inc/1068292762
11 360 DigiTech, Inc. NAS:QFIN 28.8 Medium ESG Risk https://www.sustainalytics.com/esg-rating/360-digitech-inc/2006584641
12 360 Security Technology, Inc. SHG:601360 19.7 Low ESG Risk https://www.sustainalytics.com/esg-rating/360-security-technology-inc/2005880516
13 361 Degrees International Ltd. HKG:1361 19.1 Low ESG Risk https://www.sustainalytics.com/esg-rating/361-degrees-international-ltd/1068063178
14 3D Systems Corp. NYS:DDD 25.8 Medium ESG Risk https://www.sustainalytics.com/esg-rating/3d-systems-corp/1008186648
15 3i Group PLC LON:III 11.6 Low ESG Risk https://www.sustainalytics.com/esg-rating/3i-group-plc/1007896757
16 3M Co NYS:MMM 33.6 High ESG Risk https://www.sustainalytics.com/esg-rating/3m-co/1008167440
17 3M India Ltd. BOM:523395 23.8 Medium ESG Risk https://www.sustainalytics.com/esg-rating/3m-india-ltd/1008759597
18 3R Petroleum Óleo e Gás SA BSP:RRRP3 56.0 Severe ESG Risk https://www.sustainalytics.com/esg-rating/3r-petroleum-leo-e-g-s-sa/2008351136
19 3SBio Inc HKG:1530 27.1 Medium ESG Risk https://www.sustainalytics.com/esg-rating/3sbio-inc/1032042837
EDIT: Added "Link" column to dataframe.