403 Forbidden while using postman & python

user_19570773 0
Viewed 141

Actually, I am trying to collect some stock data from API. It returns a success response of 200 while using the browser. But once I tried to call via Postman or Python script, it returns 403 FORBIDDEN.

As per my understanding so far, this API doesn't require a token or authorization because you can call the API directly from the browser.

Here's the API Url:

https://idx.co.id/umbraco/Surface/TradingSummary/GetStockSummary?Length=3&date=20220714

Here's the website:

https://idx.co.id/data-pasar/ringkasan-perdagangan/ringkasan-saham/

I have tried several ways, but it doesn't solve my issue:

  • I already put User-Agent & Accept.
  • Put authorization, put token doesn't help.
  • Curl this API also doesn't help much.

[postman]1

[curl]2

[python]3

import requests
parameters = {
    "Length": 2,
    "date": 20220714
}
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
    'accept-language': 'en-US,en;q=0.9'
}
base_url = "https://idx.co.id/umbraco/Surface/TradingSummary/GetStockSummary"
response = requests.get(url = base_url, params = parameters, headers = headers)
response
python-requests cloudflare web-scraping http-status-code-403 python
1 Answers

As per my understanding so far, this API doesn't require token or authorization because you can call the API directly from browser.

It is still possible that you require some token/cookie in the background that you need to add to your header. For example CSRF tokens if you are filling out a form.

In your browser development tools, look into the network tab and inspect the request. There you can see a cookie named __cf_bm which seems to come from Cloudflare.

Assuming that this cookie is only set on the main page, you could try to use a session to first acquire the cookie, then request the api.

import requests


session = requests.session()
main_url = 'https://idx.co.id/'
api_url = 'https://idx.co.id/umbraco/Surface/TradingSummary/GetStockSummary?Length=3&date=20220714'


# Getting the cookie
session.get(main_url)

# Requesting the api endpoint
result = requests.get(api_url)

But this still results in You do not have access to idx.co.id. The anti bot system of Cloudflare recognizes that you are not in a browser. It is possible that you need to have javascript enabled to get a valid cookie. I would recommend to use selenium to have javascript enabled.

user_105523870
Related