Python Requests 403 Forbidden using Requests IP Rotator & Random User Agent libraries

Viewed 190

I have been using:

  1. Requests IP Rotator library, to proxy traffic through AWS API Gateway, which gives a new IP for each request.
  2. And Random User Agents library to generate random user agents:
from random_user_agent.user_agent import UserAgent
from random_user_agent.params import SoftwareName, OperatingSystem
import requests
from requests_ip_rotator import ApiGateway, EXTRA_REGIONS

software_names = [SoftwareName.CHROME.value] #use only chrome
operating_systems = [OperatingSystem.WINDOWS.value, OperatingSystem.LINUX.value] # use Win and Linux   

user_agent_rotator = UserAgent(software_names=software_names, operating_systems=operating_systems, limit=5000) # request for list of 5000 user agents

# Get list of user agents.
user_agents = user_agent_rotator.get_user_agents()



gateway = ApiGateway("https://www.something.com",
                     access_key_id = 'xxxxxxxxxxxxxxxx', 
                     access_key_secret 'xxxxxxxxxxxxxxxxxxxx' )
gateway.start()

session = requests.Session()
session.mount("https://www.something.com, gateway)

for page in page_numbers[:2000]:
     user_agent = user_agent_rotator.get_random_user_agent() # Get Random User Agent String.
     response = session.get(f"https://www.something.com{page}",headers={'User- Agent':f'{user_agent}'}) # loop adding random proxy and user agent, an example of headers format = {'User-Agent': 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36,gzip(gfe)'}


gateway.shutdown()

This goes well for a few hundred requests, but eventually fails with 403 Forbidden error, I was going to ask how is it possible that sometimes 100,200,500,800 requests go through but they all fail at some point (3 digit number request).

I see in other posts people suggest changing cookies etc, but how could I determine what is causing "forbidden" and not changing everything possible in my requests. And why does it take a few hundred requests to eventually fail?

0 Answers
Related