I have been using:
- Requests IP Rotator library, to proxy traffic through AWS API Gateway, which gives a new IP for each request.
- And Random User Agents library to generate random user agents:
from random_user_agent.user_agent import UserAgent
from random_user_agent.params import SoftwareName, OperatingSystem
import requests
from requests_ip_rotator import ApiGateway, EXTRA_REGIONS
software_names = [SoftwareName.CHROME.value] #use only chrome
operating_systems = [OperatingSystem.WINDOWS.value, OperatingSystem.LINUX.value] # use Win and Linux
user_agent_rotator = UserAgent(software_names=software_names, operating_systems=operating_systems, limit=5000) # request for list of 5000 user agents
# Get list of user agents.
user_agents = user_agent_rotator.get_user_agents()
gateway = ApiGateway("https://www.something.com",
access_key_id = 'xxxxxxxxxxxxxxxx',
access_key_secret 'xxxxxxxxxxxxxxxxxxxx' )
gateway.start()
session = requests.Session()
session.mount("https://www.something.com, gateway)
for page in page_numbers[:2000]:
user_agent = user_agent_rotator.get_random_user_agent() # Get Random User Agent String.
response = session.get(f"https://www.something.com{page}",headers={'User- Agent':f'{user_agent}'}) # loop adding random proxy and user agent, an example of headers format = {'User-Agent': 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36,gzip(gfe)'}
gateway.shutdown()
This goes well for a few hundred requests, but eventually fails with 403 Forbidden error, I was going to ask how is it possible that sometimes 100,200,500,800 requests go through but they all fail at some point (3 digit number request).
I see in other posts people suggest changing cookies etc, but how could I determine what is causing "forbidden" and not changing everything possible in my requests. And why does it take a few hundred requests to eventually fail?