Python: File is not fully downloaded using requests

user_11430977 0
Viewed 770

I have problem, that my file is not being downloaded fully. It always downloads just few kb from entire file which is not what I want. Do you see problem somewhere or should I add some more logic in there?

Thank you

import os
import requests


def download(url: str, dest_folder: str):
    if not os.path.exists(dest_folder):
        os.makedirs(dest_folder)  # create folder if it does not exist

    filename = url.split('/')[-1].replace(" ", "_")  # be careful with file names
    file_path = os.path.join(dest_folder, filename)

    r = requests.get(url, stream=True)
    if r.ok:
        print("SUCCESS: Saving to", os.path.abspath(file_path))
        with open(file_path, 'wb') as f:
            for chunk in r.iter_content(chunk_size=1024 * 8):
                if chunk:
                    f.write(chunk)
                    f.flush()
                    os.fsync(f.fileno())
    else:  # HTTP status code 4XX/5XX
        print("Download failed: status code {}\n{}".format(r.status_code, r.text))

## EXECUTE ##
download("SOME FILE URL",
         dest_folder="SOME FOLDER)```
python-3.x python
1 Answers

First. Could you try shutil.copyfileobj to write the file? It should be better than f.write.

with open(file_path, 'wb') as f:
    shutil.copyfileobj(r.raw, f, length=16*1024*1024)
    f.flush()
    os.fsync(f.fileno())

Second. You can get the total file size. If your file size not match, you can re-download your file agagin.

total_content_size = int(requests.get(url, stream=True).headers['Content-Length'])

Third. This is works for me to resolve the problem. You should download the file resuming at break-points. This function will download the file from the last break-points. You can add a while loop to do this until the file size is full match.

def download_file(self, url, file_path):
    file_name = url.rsplit('/', 1)[-1]
    file_full_location = file_path + "/" + file_name

    total_content_size = int(requests.get(url, stream=True).headers['Content-Length'])
    if os.path.exists(file_full_location):
        temp_size = os.path.getsize(file_full_location)
        if total_content_size == temp_size:
            return
    else:
        temp_size = 0

    headers = {'Range': 'bytes=%d-' % temp_size}
    with requests.get(url, stream = True, headers=headers) as response:
        response.raise_for_status()
        with open(file_full_location, 'ab') as f:
            shutil.copyfileobj(response.raw, f, length=16*1024*1024)
user_92656630
Related