iCrawler is great for downloading images but how do I save the source url?

user_1839484 0
Viewed 327

iCrawler is great for downloading images. But I cannot figure out how to save the image source urls for the purpose of attribution. I plan to make slide shows with the images and include a qr code embedded in the image for attribution. I'm using creative commons reuse with modification like this:

#!/usr/bin/env python3

from icrawler.builtin import GoogleImageCrawler
import os
import time
import random
from PIL import Image
from PIL.ExifTags import TAGS
import json

wordlist = open("/home/scott/unity-projects/spelling_bee/spelling-game-python-prototype/good_words.txt").read().splitlines()
pausecounter=0
for word in wordlist: 
    
    with open('/home/scott/unity-projects/spelling_bee/spelling-game-python-prototype/data/add.json') as f:
        data = json.load(f)


    example = data[0]["meanings"][0]["definitions"][0]["example"]



    pausecounter = pausecounter + 1;

    pathish="/home/scott/unity-projects/spelling_bee/spelling-game-python-prototype/images/"
    pathish2="/home/scott/unity-projects/spelling_bee/spelling-game-python-prototype/images-compressed/"
    pathish3="/home/scott/unity-projects/spelling_bee/spelling-game-python-prototype/images-videoed/"
    if (os.path.exists(pathish3+word+'.mp4')):
        print("found word: "+word)
        time.sleep(0.033)
        continue
    searchphrase = word+" "+example
    google_Crawler = GoogleImageCrawler(storage = {'root_dir': r'images'})
    google_Crawler.crawl(keyword = searchphrase, max_num = 15, filters = dict(
    license='commercial,modify'))
    
    for number in range(1,16):
        filename=pathish+str.zfill(str(number), 6)
        newfilename=pathish+word+"_"+str.zfill(str(number), 6)
        compressedpath="/home/scott/unity-projects/spelling_bee/spelling-game-python-prototype/images-compressed/"+word+"_"+str.zfill(str(number), 6)+".jpg"
        compressedpathqmp4="/home/scott/unity-projects/spelling_bee/spelling-game-python-prototype/images-videoed/"+word+".mp4"
        print(newfilename)
        print(filename)
        if (os.path.exists(filename+'.jpg')):
            os.rename(filename+'.jpg', newfilename+'.jpg')
            os.system("convert "+newfilename+".jpg -resize 512x512> -compose Copy -gravity center -extent 512x512 -quality 98 "+compressedpath)
        if (os.path.exists(filename+'.png')):
            os.rename(filename+'.png', newfilename+'.png')
            os.system("convert "+newfilename+".png -resize 512x512> -background white -alpha remove -alpha off -compose Copy -gravity center -extent 512x512 -quality 98  "+compressedpath)
    
    os.system("ffmpeg -i images-compressed/"+word+"_%06d.jpg -vf \"scale=512:512,setpts=80*PTS\" -crf 38 -profile:v high -level:v 5.1 -pix_fmt yuv420p "+compressedpathqmp4)
    time.sleep(random.random()*pausecounter)
    if pausecounter > 10:
        pausecounter = 0

For each image saved I'd like to save a file containing its source url in a separate file.

web-crawler python
1 Answers

I hacked some of the examples of the documentation. I'm not a python expert so I can't explain how this works but this is how I solved my problem. Hopefully you can follow along with the variable names. As I'm not real sure how this works but it does. The only flaw is that it spits out an additional url file.

#!/usr/bin/env python3

import base64
from icrawler import ImageDownloader
from icrawler.builtin import GoogleImageCrawler
from six.moves.urllib.parse import urlparse

class getAttribution(ImageDownloader):
    def get_filename(self, task, default_ext):
        global word
        url = urlparse(task['file_url'])[0] + "://" +urlparse(task['file_url'])[1] + urlparse(task['file_url'])[2]
        filename = super(getAttribution, self).get_filename(
        task, default_ext)
        print(url +" "+ filename)
        file = open("test-delete/"+word +"_"+ filename+".txt","w")
        file.write(url)
        file.close()
        return word +"_"+ filename

google_crawler = GoogleImageCrawler(
downloader_cls=getAttribution,
downloader_threads=1,
storage={'root_dir': 'test-delete'})

word='sand'`enter code here`

google_crawler.crawl(word, max_num=12, filters = dict(
    license='commercial,modify'))
user_18394840
Related