I try proxy by requests like below:
proxies = {
"http": "http://127.0.0.1:1080",
"https": "http://127.0.0.1:1080"
}
resp=requests.get(url,headers=headers,proxies=proxies)
with open("resp.html","w", encoding='UTF-8') as f:
f.write(resp.text)
But I try to use the proxies in scrapy:
yield scrapy.Request(
url=link,
method='GET',
headers=self.headers,
callback=self.parse_detail,
meta={"MyItem":item,"proxy": "https://127.0.0.1:1080"})
Scrapy always timeout! How can I set proxy in scrapy? Thanks!