How can I scrape a website that does not show any HTML codes in the source using Python without Selenium

Question

I would like to scrape a website that does not have any HTML codes in the page source using Python.

I have tried doing it with Selenium but I am trying to do it without Selenium as I have some difficulties importing this function to my Rasperry Pi.

https://www.cea.gov.sg/aceas/public-register/sales/1?page=1&pageSize=10&sortAscFlag=true&sort=name®istrationNumber=R

This is something that I am interested in scraping but I can't seem to do it effectively. I have tried using bs4 and requests to work with it, but there's no HTML codes for me to work with and I can't seem to find other libraries that can do it without Selenium.

import requests

r = requests.get('https://www.cea.gov.sg/aceas/public-register/sales/1?page=1&pageSize=10&sortAscFlag=true&sort=name®istrationNumber=R')
print(r.text)

This is a simplified version of what I have tried before.

D:\Codes\venv\Scripts\python.exe D:/Codes/requests_test.py 
ACEAS


Process finished with exit code 0

Barry the Platipus · Answer

In hope that your next question(s) will contain a minimal reproducible example, here is one way to scrape that information: data is being hydrated into page dynamically, via XHR calls to an API. You can see that by inspecting Dev tools - Network tab in your browser.

import requests
import pandas as pd
from tqdm import tqdm

big_df = pd.DataFrame()

headers = {
    'Origin': 'https://www.cea.gov.sg',
    'Content-Type': 'application/json;charset=UTF-8',
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36'
}

s = requests.Session()
s.headers.update(headers)
url = 'https://www.cea.gov.sg/aceas/api/internet/profile/v2/public-register/filter'

for x in tqdm(range(1, 10)):
    payload = '{"page":' + str(x) + ',"pageSize":100,"sortAscFlag":"true","registrationNumber":"R","sort":"name","profileType":2}'
    r = s.post(url, data=payload)
    df = pd.json_normalize(r.json()['data'])
    big_df = pd.concat([big_df, df], axis=0, ignore_index=True)
print(big_df)

This will display the datadframe in terminal:

id  name    businessName    licenseNumber   validityDateStart   validityDateEnd awards  disciplinaryActions registrationNumber  photoUrl    currentEa
0   0679f4a5-ca99-4c6c-a4c4-12528ece6294    'AFFAN BIN ASHAK HARI   'AFFAN A.H. L3002382K   2019-02-27T00:00:00+08:00   2022-12-31T23:59:59.99+08:00    None    None    R060832J    None    ERA REALTY NETWORK PTE LTD
1   c55b6be6-15fa-490c-a688-745a91839596    AARON BAN QI WEI    AARON BAN   L3002382K   2019-08-30T00:00:00+08:00   2022-12-31T23:59:59.99+08:00    None    None    R061593I    None    ERA REALTY NETWORK PTE LTD
2   6525dc35-5d0b-467f-8fc1-68894884e3fb    AARON GOH JIN HAO   None    L3008022J   2019-09-16T00:00:00+08:00   2022-12-31T19:14:00+08:00   None    None    R052117I    None    PROPNEX REALTY PTE. LTD.
3   38d64cfc-5cb2-4027-9add-f01ee4ed8769    AARON HUAN SHEN LI  AARON HUAN  L3008022J   2019-01-01T00:00:00+08:00   2022-12-31T14:57:00+08:00   None    None    R041988I    None    PROPNEX REALTY PTE. LTD.
4   15aa903c-2402-4bed-87d2-ef6fce88a502    AARON LEONG JIA SHENG   None    L3008022J   2021-01-01T00:00:00+08:00   2022-12-31T23:59:59.99+08:00    None    None    R062835F    None    PROPNEX REALTY PTE. LTD.
... ... ... ... ... ... ... ... ... ... ... ...
895 d845c8a7-0713-40ba-b0d9-7f06933997ee    ANG YAM NEE, AGNES JAEL JAEL ANG    L3002382K   2017-08-28T00:00:00+08:00   2022-12-31T20:14:00+08:00   None    None    R058715C    None    ERA REALTY NETWORK PTE LTD
896 705a51a0-2024-4fc3-9a84-27530a579681    ANG YAN BRYAN ANG   L3008022J   2018-02-27T00:00:00+08:00   2022-12-31T23:59:59.99+08:00    None    None    R009088G    None    PROPNEX REALTY PTE. LTD.
897 968a17a8-b0bb-455f-a95b-0281084d1da5    ANG YANG MING   ANG YM  L3008022J   2011-01-01T00:00:00+08:00   2022-12-31T20:24:00+08:00   None    None    R009471H    None    PROPNEX REALTY PTE. LTD.
898 98fbdf7a-107a-4980-a7b4-82560123769e    ANG YAP CHOW    Ang Yap Chow    L3008899K   2021-04-30T00:00:00+08:00   2022-12-31T00:21:00+08:00   None    None    R063534D    None    HUTTONS ASIA PTE. LTD.
899 848efbcf-3050-4a1d-9394-bf4e8051f4a0    ANG YAP HWEE    SUNNY   L3010497H   2018-01-01T00:00:00+08:00   2022-12-31T11:44:00+08:00   None    None    R040155F    None    ASSET PROPERTY PRIVATE LIMITED

There are 341 pages, and you can go through all, the example above is only pulling the first 10 pages of data.

Relevant pandas documentation: https://pandas.pydata.org/pandas-docs/stable/index.html

Requests documentation: https://requests.readthedocs.io/en/latest/

How can I scrape a website that does not show any HTML codes in the source using Python without Selenium

1 Answers