I'm trying to download a PDF file using the requests module, the code is as below:
import requests
url = "<url of the pdf>"
r = requests.get(url, stream=True, timeout=(60, 120), headers={'Connection': 'keep-alive','User-Agent': 'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.10136'})
print(r.headers)
print(r.status_code)
try:
with open('blah.pdf', 'wb') as f:
for chunk in r:
# print(chunk)
f.write(chunk)
except Exception as e:
print(e)
Output is given below:
{'Cache-Control': 'private', 'Transfer-Encoding': 'chunked', 'Content-Type': 'application/pdf', 'Server': 'Microsoft-IIS/7.5', 'X-AspNet-Version': '4.0.30319', 'X-Powered-By': 'ASP.NET', 'Date': 'Wed, 02 Oct 2019 05:17:11 GMT', 'Set-Cookie': 'bbb=rd102o00000000000000000000ffff978433aao80; path=/; Httponly; Secure'}
200
('Connection broken: IncompleteRead(0 bytes read, 2 more expected)', IncompleteRead(0 bytes read, 2 more expected))
Here is the full stack trace:
Traceback (most recent call last):
File "/storage/anaconda3/lib/python3.7/site-packages/urllib3/response.py", line 425, in _error_catcher
yield
File "/storage/anaconda3/lib/python3.7/site-packages/urllib3/response.py", line 755, in read_chunked
chunk = self._handle_chunk(amt)
File "/storage/anaconda3/lib/python3.7/site-packages/urllib3/response.py", line 709, in _handle_chunk
self._fp._safe_read(2) # Toss the CRLF at the end of the chunk.
File "/storage/anaconda3/lib/python3.7/http/client.py", line 612, in _safe_read
raise IncompleteRead(b''.join(s), amt)
http.client.IncompleteRead: IncompleteRead(0 bytes read, 2 more expected)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/storage/anaconda3/lib/python3.7/site-packages/requests/models.py", line 750, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/storage/anaconda3/lib/python3.7/site-packages/urllib3/response.py", line 560, in stream
for line in self.read_chunked(amt, decode_content=decode_content):
File "/storage/anaconda3/lib/python3.7/site-packages/urllib3/response.py", line 781, in read_chunked
self._original_response.close()
File "/storage/anaconda3/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/storage/anaconda3/lib/python3.7/site-packages/urllib3/response.py", line 443, in _error_catcher
raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read, 2 more expected)', IncompleteRead(0 bytes read, 2 more expected))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test.py", line 12, in <module>
for chunk in r:
File "/storage/anaconda3/lib/python3.7/site-packages/requests/models.py", line 753, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read, 2 more expected)', IncompleteRead(0 bytes read, 2 more expected))
When I open that pdf on Web Browsers such as Google Chrome, chrome's builtin pdf plugin can load it properly and it is possible to read on the browser. However, If I try to download it by clicking on the download icon I get Failed - Network Error Firefox can't load/download it. (Both Firefox & Chrome are upgraded to latest version) When I test it on a windows machine Microsoft edge was able to download the pdf though...
The above code, If I test it with some other pdfs such as this one:
https://adobe.com/content/dam/acom/en/accessibility/products/acrobat/pdfs/acrobat-x-accessibility-checker.pdf
It works perfectly.
I've tried some command-line tools such as curl, wget, aria2c (with proper headers set like browser request) all failed to download the pdf.
wget output:
connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/pdf]
Saving to: ‘blah.pdf’
<pdf_url> [ <=> ] 101.68K 66.1KB/s in 1.5s
2019-10-02 11:29:50 (69.1 KB/s) - Read error at byte 108786 (Success).
The file downloaded using wget is corrupted.
Another thing that I've tried is to inspect it using mitm and chromedriver+selenium combination.
The automated chrome browser can't load the pdf and shows an error:
502 Bad Gateway
HttpSyntaxException('Malformed chunked body',)
How can I download this pdf using requests module? Any help will be very much appreciated.