Automate The Boring Stuff - Image Site Downloader

user_11192394 0
Viewed 809

I am writing a project from the Automate The Boring Stuff book. The task is the following:

Image Site Downloader

Write a program that goes to a photo-sharing site like Flickr or Imgur, searches for a category of photos, and then downloads all the resulting images. You could write a program that works with any photo site that has a search feature.

Here is my code:

import requests, bs4, os

# The outerHTML file which I got by rightClicking and copying the <html> tag on 'page source'
flickrFile=open('flickrHtml.html',encoding="utf8")

#Parsing the HTML document 
flickrSoup=bs4.BeautifulSoup(flickrFile,'html.parser')

# categoryElem is the Element which has image source inside
categoryElem=flickrSoup.select("a[class='overlay']")
#len(categoryElem)=849

os.makedirs('FlickrImages', exist_ok=True) 
for i in range(len(categoryElem)-1):
    
    # Regex searching for the href
    import re
    html=str(categoryElem[i])
    htmlRegex=re.compile(r'href.*/"')
    mo=htmlRegex.search(html)
    imageUrl=mo.group()

    imageUrl=imageUrl.replace('"','')
    imageUrl=imageUrl.replace('href=','')

    imageUrlFlickr="https://www.flickr.com"+str(imageUrl)

    # Downloading the response object of the Image URL
    res = requests.get(imageUrlFlickr)
    imageSoup=bs4.BeautifulSoup(res.text)
    picElem=imageSoup.select('div[class="view photo-well-media-scrappy-view requiredToShowOnServer"] img')

    # Regex searching for the jpg file in the picElem HTML element
    html=str(picElem)
    htmlRegex=re.compile(r'//live.*\.jpg')
    mo=htmlRegex.search(html)
    try:
        imageUrlRegex=mo.group()
    except Exception as exc:
        print('There was a problem: %s' % (exc))
    res1=requests.get('https:'+imageUrlRegex)
    try:
        res1.raise_for_status()
    except Exception as exc:
        print('There was a problem: %s' % (exc))  
    # Dowloading the jpg to my folder
    imageFile = open(os.path.join('FlickrImages', os.path.basename(imageUrlRegex)), 'wb')
    for chunk in res1.iter_content(100000):
        imageFile.write(chunk)

After looking up this question, I figured that for downloading all of the 4 million results for picture "Sea", I copy (as said in the answer to the question stated) the whole OuterHTML. If I would have not looked at this question, and not have copied the full HTML source (in my code, it's stored in flickrFile=open('flickrHtml.html',encoding="utf8")), I would end up having categoryElem equal to 24, and hence downloading only 24 pictures, instead of 849 pics.

There are 4 million pictures, how do I download all of them, without having to download the HTML source to a separate file?

I was thinking of my program to do the following:

  1. Getting the url of the first picture of the search --> downloading the picture --> getting the next picture's url --> downloading the pic.... and so on until there is nothing left to download.

I did not go with the first approach because I did not know how to get to the link of the first picture. I tried getting the URL of it, but then when I inspect the element of the first pic (or any other pic) from the "photo stream", it gives me a link to the "photo stream" of the specific user, not the general "Sea Search photo stream".

Here is the link for the photo stream Search

If someone could help me with that too, it would be fantastic.

Here is some code from someone who did the same task, but he's downloading only the first 24 pictures, which are the pictures that show up on the original, un-rendered HTML

beautifulsoup url python
2 Answers

If you want to use requests + Beautfulsoup, Try this below(by passing argument page):

import re, requests, threading, os
from bs4 import BeautifulSoup

def download_image(url):
    with open(os.path.basename(url), "wb") as f:
        f.write(requests.get(url).content)
    print(url, "download successfully")

original_url = "https://www.flickr.com/search/?text=sea&view_all=1&page={}"

pages = range(1, 5000) # not sure how many pages here

for page in pages:
    concat_url = original_url.format(page)
    print("Now it is page", page)
    soup = BeautifulSoup(requests.get(concat_url).content, "lxml")
    soup_list = soup.select(".photo-list-photo-view")
    for element in soup_list:
        img_url = 'https:'+re.search(r'url\((.*)\)', element.get("style")).group(1)
        # the url like: https://live.staticflickr.com/xxx/xxxxx_m.jpg
        # if you want to get a clearer(and larger) picture, remove the "_m" in the end of the url.
        # For prevent IO block,I create a thread to download it.pass the url of the image as argument.
        threading.Thread(target=download_image, args=(img_url,)).start()

If using selenium, it could be easier, example code like:

from selenium import webdriver
import re, requests, threading, os

# download_image
def download_image(url):
    with open(os.path.basename(url), "wb") as f:
        f.write(requests.get(url).content)


driver = webdriver.Chrome()
original_url = "https://www.flickr.com/search/?text=sea&view_all=1&page={}"

pages = range(1, 5000) # not sure how many pages here

for page in pages:
    concat_url = original_url.format(page)
    print("Now it is page", page)
    driver.get(concat_url)
    for element in driver.find_elements_by_css_selector(".photo-list-photo-view"):
        img_url = 'https:'+re.search(r'url\(\"(.*)\"\)', element.get_attribute("style")).group(1)
        # the url like: https://live.staticflickr.com/xxx/xxxxx_m.jpg
        # if you want to get a clearer(and larger) picture, remove the "_m" in the end of the url.
        # For prevent IO block,I create a thread to download it.pass the url of the image as argument.
        threading.Thread(target=download_image, args=(img_url, )).start()

And it download successfully on my PC.

enter image description here

user_118112550

First off - scraping 4 million results from a website like Flicker is likely to be unethical. Web scrapers should do their best to respect the website from which they are scraping by minimizing their load on servers. 4 million requests in a short amount of time is likely to get your IP banned. If you used proxies you could get around this but again - highly unethical. You also run into the risk of copyright issues since a lot of the images on flicker are subject to copyright.

If you were to go about doing this you would have to use Scrapy and possibly a Scrapy-Selenium combo. Scrapy is great for running concurrent requests meaning you can request a large number of images at the same time. You can learn more about Scrapy here:https://docs.scrapy.org/en/latest/

The workflow would look something like this:

  1. Scrapy makes a request to the website for the html - parse through it to find all tags with class='overlay no-outline'
  2. Scrapy makes a request to each url concurrently. This means that the urls won't be followed one by one but instead side by side.
  3. As the images are returned they get added to your database/storage space
  4. Scrapy (maybe Selenium) scrolls the infinitely scrolling page and repeats without iterating over already checked images (keep index of last scanned item).

This is what Scrapy would entail but I strongly recommend not attempting to scrape 4 million elements. You would probably find that the performance issues you run into would not be worth your time especially since this is supposed to be a learning experience and you will likely never have to scrape that many elements.

user_135994670
Related