Update: The site is responding to command line curl but not to Python requests.
I have scraped many websites and extracted 100s of GBs of data but never faced a problem like this. I am not able to understand why the site is throwing "104, connection reset by peer" error. What am I missing?
from requests import Session
link = 'http://103.94.204.78/searchengine/searchen.aspx'
headers = {
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'Accept-Encoding': 'gzip, deflate',
'Accept-Language': 'en-GB,en-US;q=0.9,en;q=0.8',
'Cache-Control': 'max-age=0',
'Connection': 'keep-alive',
'DNT': '1',
'Host': '103.94.204.78',
'Upgrade-Insecure-Requests': '1',
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36 OPR/80.0.4170.72'
}
with Session() as session:
resp = session.post(url=link, headers=headers)
print(resp.text)
Output:
Traceback (most recent call last):
File "/home/naveed/myenv/lib/python3.8/site-packages/urllib3/response.py", line 438, in _error_catcher
yield
File "/home/naveed/myenv/lib/python3.8/site-packages/urllib3/response.py", line 519, in read
data = self._fp.read(amt) if not fp_closed else b""
File "/usr/lib/python3.8/http/client.py", line 455, in read
n = self.readinto(b)
File "/usr/lib/python3.8/http/client.py", line 499, in readinto
n = self.fp.readinto(b)
File "/usr/lib/python3.8/socket.py", line 669, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/naveed/myenv/lib/python3.8/site-packages/requests/models.py", line 758, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/home/naveed/myenv/lib/python3.8/site-packages/urllib3/response.py", line 576, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/home/naveed/myenv/lib/python3.8/site-packages/urllib3/response.py", line 541, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "/home/naveed/myenv/lib/python3.8/site-packages/urllib3/response.py", line 455, in _error_catcher
raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./preproc.py", line 36, in <module>
resp = session.get(url=link, headers=headers)
File "/home/naveed/myenv/lib/python3.8/site-packages/requests/sessions.py", line 555, in get
return self.request('GET', url, **kwargs)
File "/home/naveed/myenv/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/home/naveed/myenv/lib/python3.8/site-packages/requests/sessions.py", line 697, in send
r.content
File "/home/naveed/myenv/lib/python3.8/site-packages/requests/models.py", line 836, in content
self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
File "/home/naveed/myenv/lib/python3.8/site-packages/requests/models.py", line 761, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
I have installed pyopenssl ndg-httpsclient pyasn1. My requests version is 2.26.0