CRAWL QUESTION: a question abount requests

user_10345077 0
Viewed 40

I am trying to set up random proxies when using the request, and I have encountered some problems.this is my code:

import requests
import random
pool = ['220.186.175.252:4216','106.110.39.106:4232']
proxy={'https':random.choice(pool)}
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'}
test_url = "http://httpbin.org/ip" # a url test ip
response = requests.get(url=test_url,headers=headers,proxies=proxy)
text = response.text
print(text)

and the result:

{"origin": "112.10.164.203"}

it doesn't work, so I try to change my proxy, I thought maybe it use http instead of https, I change proxy to this:

proxy={'https':random.choice(pool)}

and unfortunately I get a ERROR:

requests.exceptions.ProxyError: HTTPConnectionPool(host='106.110.39.106', port=4232): Max retries exceeded with url: http://httpbin.org/ip (Caused by ProxyError('Cannot connect to proxy.', RemoteDisconnected('Remote end closed connection without response',)))

SO, I have two questions: 1.How to set the random proxy in requests 2.why i change the proxy get this ERROR

i am so appricate if you can solve my problems!

python-requests
1 Answers

You did right, why you got this error is that your proxy does not support http request. You need to know what kind of protocol does it support before you use it. free-proxy-list

And this is the way i define a random proxy 

import requests
import random
https = ['220.186.175.252:4216','106.110.39.106:4232']
http = ["169.50.180.250:3128"]
proxy={'https':random.choice(https),"http":random.choice(http)}
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'}
test_url = "http://httpbin.org/ip" # a url test ip
response = requests.get(url=test_url,headers=headers,proxies=proxy)
text = response.text
print(text)
user_93806940
Related