requests.get on large file, incomplete download but can't find indication of error

user_1013719 0
Viewed 1152

I'm trying to use requests to download a large file (namely an android cts zip file) using the same technique as this answer. Intermittently it is failing to download the whole file, but I can't find any indication that something has gone wrong until I try to unzip the file

CTS_URL = 'http://dl.google.com/dl/android/cts/android-cts-8.0_r14-linux_x86-x86.zip'
CTS_ZIP = 'android-cts-8.0_r14-linux_x86-x86.zip'

import requests

req = requests.get(CTS_URL, stream=True)
with open(CTS_ZIP, 'wb') as cts_zip_file:
  for chunk in req.iter_content(chunk_size=4096):
    cts_zip_file.write(chunk)

later when I try to unzip I'm getting a BadZipFile("File is not a zip file") error, because the file hasn't been fully-downloaded

import zipfile
zipfile.ZipFile(CTS_ZIP)  # fails

However, I can't get any indication from the request object that something has gone wrong. req.status is 200, req.ok is True.

Does req know that something went wrong? I currently have one of these request objects in an interactive prompt so I can inspect it further.

python-requests python-3.x python
1 Answers

The code you share worked for me. However, you might consider about following things:

  1. Like sraw mentioned, check Content-length value.
  2. Check the file you downloaded. If the size of file is correct, you can try to extract the file manually in shell (just wanted to make sure).
  3. You can try to use urllib (example)
  4. It could be also server problem as well -- just in case. You can try to download different file.
user_20556900
Related