Download images with python Connection broken

user_16329544 0
Viewed 128

i'm trying to download 200k images using their URL. This is my code:

import requests # to get image from the web
import shutil # to save it locally
r = requests.get(image_url, stream = True)

# Check if the image was retrieved successfully
if r.status_code == 200:
    # Set decode_content value to True, otherwise the downloaded image file's size will be zero.
    r.raw.decode_content = True
    
    if not os.path.isdir('images/' + filename.rsplit('/',1)[0] + '/'):
        os.makedirs('images/' + filename.rsplit('/',1)[0] + '/')

    with open('images/' + filename,'wb') as f:
        shutil.copyfileobj(r.raw, f)

The when i run it, it downloads some images but the rest doesn't. It gives the error:

urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead

I have no idea why or when this happens. Maybe when a URL is unreachable? How can I assure that everything UP will be downloaded and exceptions will be passed?

request python
1 Answers

What about using a try/except?

import requests # to get image from the web
import shutil # to save it locally
try:
    r = requests.get(image_url, stream = True)

    # Check if the image was retrieved successfully
    if r.status_code == 200:
        # Set decode_content value to True, otherwise the downloaded image file's size will be zero.
        r.raw.decode_content = True
        
        if not os.path.isdir('images/' + filename.rsplit('/',1)[0] + '/'):
            os.makedirs('images/' + filename.rsplit('/',1)[0] + '/')

        with open('images/' + filename,'wb') as f:
            shutil.copyfileobj(r.raw, f)
except urllib3.exceptions.ProtocolError as error:
    print("skipped error: " + error)

Perhaps to download such a large number of images you would be interested in an asynchronous web framework like aiohttp. This would save you from having to wait for a slow site to send you its image to download more.

user_101449630
Related