403 Forbidden on requests.post through Python although google chrome receiving response

Viewed 2187

I am very new to web-scraping. I'm trying to fetch data of flipkart sellers for a product by reverse-engineering the requests and scrape the data. The URL I'm trying on is : https://www.flipkart.com/sellers?pid=HSAFFJWT2WUYHHHX, which contains list of sellers for a particular product.

When I open the network tab in Google chrome I can see that its making a POST request to: https://www.flipkart.com/api/3/page/dynamic/product-sellers with request payload as : {"requestContext":{"productId":"HSAFFJWT2WUYHHHX"},"locationContext":{"pincode":"110092"}}

I am copying the request headers and the payload, the browser has sent, in my code but I'm getting a 403 Forbidden message instead of getting a valid JSON data.

Alternately, I've been successful in using Selenium to open this page (https://www.flipkart.com/sellers?pid=HSAFFJWT2WUYHHHX) and then parse the sellers data using Beautiful Soup . However, selenium is very very slow. That is why I'm trying to fetch the json directly which populates the sellers data dynamically on this page. I've tried making a post request from my code as well as Postman but with no luck.

import requests
import json
headers = {
            'Accept': '*/*',
            'Accept-Encoding': 'gzip, deflate, br',
            'Accept-Language': 'en-US,en;q=0.9',
            'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
            'Connection': 'keep-alive',
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',
            'Referer' :   'https://www.flipkart.com/sellers?pid=HSAFFJWT2WUYHHHX',
            'Content-Type': 'application/json',
            'Origin': 'https://www.flipkart.com',
            'Host': 'www.flipkart.com',
            'Pragma': 'no-cache'
            }

url = "https://www.flipkart.com/api/3/page/dynamic/product-sellers"
payLoad = {"requestContext":{"productId":"HSAFFJWT2WUYHHHX"},"locationContext":{"pincode":"001195"}}
response = requests.post(url,headers = headers,data = json.dumps(payLoad))
print(response.status_code,response.reason)
print(response.text)
print(response.request.headers)

I'm expecting to get a Json like this programmatically : response

But I'm getting 403 Forbidden. Even though the page does not require a user to login.

Perhaps there is some security token that the api expects that I have not given which is why its throwing 403 Forbidden.

Can anyone help me figure out what am I missing to get the Json?

Thanks!

1 Answers

Added in the x-user-agent. Give this a shot:

import requests
import json
headers = {
            'Accept': '*/*',
            'Accept-Encoding': 'gzip, deflate, br',
            'Accept-Language': 'en-US,en;q=0.9',
            'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
            'Connection': 'keep-alive',
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',
            'X-user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36 FKUA/website/41/website/Desktop',
            'Referer' :   'https://www.flipkart.com/sellers?pid=HSAFFJWT2WUYHHHX',
            'Content-Type': 'application/json',
            'Origin': 'https://www.flipkart.com',
            'Host': 'www.flipkart.com',
            'Pragma': 'no-cache'
            }

url = "https://www.flipkart.com/api/3/page/dynamic/product-sellers"
payLoad = {"requestContext":{"productId":"HSAFFJWT2WUYHHHX"},"locationContext":{"pincode":"001195"}}
response = requests.post(url,headers = headers,data = json.dumps(payLoad))
print(response.status_code,response.reason)
print(response.text)
print(response.request.headers)

Output:

print(response.status_code,response.reason)
200 OK
Related