Python requests best solution to speed up web scraper with rotating residential proxy

Viewed 27

I currently have a web scraper using Python Requests and ThreadPoolExecutor with a read timeout of 5 seconds.
The problem is that I'm using a rotating residential proxy provider, and since it's a residential proxy, the speed can be either incredibly slow, or fast. The ip address changes every request, but it's connecting to one ip address, causing me to not be able to choose the ip address and filter out the slow ones.
Since the speed can be either incredibly slow or fast, and the read timeout is set to 5 seconds, the amount of pages scraped every minute is usually pretty low since the a lot of the threads wait for 5 seconds before giving up, however, if I put it at, let's say, 2 seconds, a lot of the requests will fail even though the proxy speed is okay since it's not that fast.
What's the best way to go about this?

0 Answers
Related