Downloading 1000s of images using php , how to bypass blank images

user_2200854 0
Viewed 324

I have a php file that is creating a list of url paths of images i want to download. Currently i have just been creating a list , copying and pasting it in a desktop app that will download images by url list in txt file. I wanted to go ahead and add the download to the php file to bypass that current step.

currently the following generates a list of urls i use to copy and paste to desktop downloader app

foreach ($espn_ar as $key => $value) echo 'ht'.'tps://a.espncdn.com/i/headshots/nfl/players/full/'.$espn_ar[$key].'.png<br>';

the list that is generated looks like this , with approximately 2500 url paths

https://a.espncdn.com/i/headshots/nfl/players/full/2580.png
https://a.espncdn.com/i/headshots/nfl/players/full/2330.png
https://a.espncdn.com/i/headshots/nfl/players/full/2977742.png
https://a.espncdn.com/i/headshots/nfl/players/full/5528.png
https://a.espncdn.com/i/headshots/nfl/players/full/5529.png
https://a.espncdn.com/i/headshots/nfl/players/full/5536.png
https://a.espncdn.com/i/headshots/nfl/players/full/5713.png
https://a.espncdn.com/i/headshots/nfl/players/full/8439.png
https://a.espncdn.com/i/headshots/nfl/players/full/8461.png
https://a.espncdn.com/i/headshots/nfl/players/full/8479.png

To bypass the step of copying and pasting to a desktop app , i added the following

foreach ($espn_ar as $key => $value) {
    $url = 'ht' . 'tps://a.espncdn.com/i/headshots/nfl/players/full/' . $espn_ar[$key] . '.png';
    $img = 'images/' . $espn_ar[$key] . '.png';
    $ch = curl_init($url);
    $fp = fopen($img, 'wb');
    curl_setopt($ch, CURLOPT_FILE, $fp);
    curl_setopt($ch, CURLOPT_HEADER, 0);
    curl_exec($ch);
    curl_close($ch);
    fclose($fp);
}

This seems to be working to download all images into a directory setup called "images"

Question 1 - is what i have done the best method for what i'm trying to achieve ?

Question 2 - when using the desktop app it doesn't download a blank image if the url path doesn't contain an image, however using the php method it does download a blank image if one of the url paths doesn't contain any image, so how can i add something to not download a blank png file, or only download files over 1kb, or delete all files 1kb or smaller after all files downloaded, not sure what is best way to handle that.

php
1 Answers

is what i have done the best method for what i'm trying to achieve ?

Answers to such kind of questions are generally based on your explanation of 'best'. Since you said you have very little knowledge of php, I just guess you want a simple approach, and the simplest approach I can find is:

file_put_contents($img, file_get_contents($url));

Notice that you should set allow_url_fopen = On and enable openssl (extension=openssl) in php.ini.

Here is a full example:

foreach ($espn_ar as $value) {
    $url = "https://a.espncdn.com/i/headshots/nfl/players/full/$value.png";
    $img = "images/$value.png";
    $content = file_get_contents($url);
    file_put_contents($img, $content);
}

when using the desktop app it doesn't download a blank image if the url path doesn't contain an image, however using the php method it does download a blank image if one of the url paths doesn't contain any image, so how can i add something to not download a blank png file, or only download files over 1kb , or delete all files 1kb or smaller after all files downloaded, not sure what is best way to handle that.

Because I can't check all the URLs, I guess these blank images may be caused by 404 error, or the images are really blank. I'll check them all.

    $content = file_get_contents($url);
    
    # download failed
    if($content === false)
        continue;
    
    # 404 error
    if(strpos($http_response_header[0], '404 Not Found') !== false)
        continue;

    # size < 1kb
    if(strlen($content) < 1024)
        continue;

    file_put_contents($img, $content);
user_61965680
Related