Python requests returns 404 error status from today on amazon "sold by" page

Admin 1
Viewed 176

There

https://www.amazon.de/sp?marketplaceID=A1PA6795UKMFR9&seller=A135E02VGPPVQ&isAmazonFulfilled=1&ref=dp_merchant_link

I just can access on browser as well, but python requests returns 404 error status.

Until yesterday, this page worked as well with python requests. but from today, it does not work for me and it returns 404 error status.

import requests

headers = {
    'Connection': 'keep-alive',
    'rtt': '300',
    'downlink': '0.4',
    'ect': '3g',
    'sec-ch-ua': '" Not;A Brand";v="99", "Google Chrome";v="97", "Chromium";v="97"',
    'sec-ch-ua-mobile': '?0',
    'sec-ch-ua-platform': '"Windows"',
    'Upgrade-Insecure-Requests': '1',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
    'Sec-Fetch-Site': 'none',
    'Sec-Fetch-Mode': 'navigate',
    'Sec-Fetch-User': '?1',
    'Sec-Fetch-Dest': 'document',
    'Accept-Language': 'en-US,en;q=0.9,ko;q=0.8',
}

response = requests.get(
    'https://www.amazon.de/sp?marketplaceID=A1PA6795UKMFR9&seller=A135E02VGPPVQ&isAmazonFulfilled=1&ref=dp_merchant_link',
    headers=headers
)
print(response.status_code) # 404

I really will appreciate if I can get any help from you.

Regards.

python-requests http-status-code-404 python
2 Answers

It works just fine to me. Maybe you can try changing your user agent.

import requests
from fake_useragent import UserAgent # fake user agent library

# random user-agent
ua = UserAgent()
user_agent = ua.random

headers = {
    'Connection': 'keep-alive',
    'rtt': '300',
    'downlink': '0.4',
    'ect': '3g',
    'sec-ch-ua': '" Not;A Brand";v="99", "Google Chrome";v="97", "Chromium";v="97"',
    'sec-ch-ua-mobile': '?0',
    'sec-ch-ua-platform': '"Windows"',
    'Upgrade-Insecure-Requests': '1',
    'User-Agent': user_agent,
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
    'Sec-Fetch-Site': 'none',
    'Sec-Fetch-Mode': 'navigate',
    'Sec-Fetch-User': '?1',
    'Sec-Fetch-Dest': 'document',
    'Accept-Language': 'en-US,en;q=0.9,ko;q=0.8',
}

response = requests.get(
    'https://www.amazon.de/sp?marketplaceID=A1PA6795UKMFR9&seller=A135E02VGPPVQ&isAmazonFulfilled=1&ref=dp_merchant_link',
     headers=headers
 )

print(response.status_code)
user_107096820

I suggest you add a referer url in header and generally I don't think Amazon allow scraping but use selenium if you want to scrape it easily (sure it may be overkill) but has more customization

user_176357250
Related