I always get timeout error when I try to do "scrapy fetch https://www.sunnysports.com/robots.txt"
Error message
DEBUG: Retrying <GET https://www.sunnysports.com/robots.txt> (failed 2 times): User timeout caused connection failure: Getting https://www.sunnysports.com/robots.txt took longer than 180.0 seconds..
But I can get the content with curl -v or urllib2. I tried to align the request header ex. let scrapy request header be the same as curl or let curl request header be the same as scrapy. Curl always works but scrapy always fail.
Python2.7 test code
import urllib2
req = urllib2.Request('https://www.sunnysports.com/robots.txt')
response = urllib2.urlopen(req)
the_page = response.read()
My scrapy version
$scrapy version -v
Scrapy : 2.1.0
lxml : 4.5.1.0
libxml2 : 2.9.10
cssselect : 1.1.0
parsel : 1.6.0
w3lib : 1.22.0
Twisted : 20.3.0
Python : 3.6.0 (v3.6.0:41df79263a11, Dec 22 2016, 17:23:13) - [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
pyOpenSSL : 19.1.0 (OpenSSL 1.1.1g 21 Apr 2020)
cryptography : 2.9.2
Platform : Darwin-18.7.0-x86_64-i386-64bit