How to download an image from a website every six minutes?

user_14140072 0
Viewed 93

I am working on a machine learning project and need a LOT of pictures for the data set that will train my program. The website https://its.txdot.gov/ITS_WEB/FrontEnd/default.html?r=SAT&p=San%20Antonio&t=cctv has pictures that are updated every six minutes. I need to save the image at LP 1604 at Kyle Seal Pkwy, but can't figure out how. I'm trying to right click on the image using action chains to save the image. Here's what I have so far:

driver.get('https://its.txdot.gov/ITS_WEB/FrontEnd/default.html?r=SAT&p=San%20Antonio&t=cctv')
time.sleep(5) #to let the site load
driver.find_element_by_id('LP-1604').click() #to get to the 1604 tab
time.sleep(5) #to let the site load
pic = driver.find_element_by_id('LP 1604 at Kyle Seale Pkwy__SAT')
action = ActionChains(driver)
action.context_click(pic)

The drop-down menu that usually pops up when you right-click is not showing up. And I feel like there has to be a better way to do this than right-click. I know how to wrap this in a loop that will execute every six minutes, so I don't need help there. It's just the downloading the image part. One of the problems I run into is that all the images are under the same url and most examples out there use urls. Any suggestions would be helpful.

selenium python
2 Answers

I think that it could be help you do save the images in your Pc:

from PIL import Image

def save_image_on_disk(driver, element, path):
    location = element.location
    size = element.size
    # saves screenshot of entire page
    driver.save_screenshot(path)

    # uses PIL library to open image in memory
    image = Image.open(path)

    left = location['x']
    top = location['y'] + 0
    right = location['x'] + size['width']
    bottom = location['y'] + size['height'] + 0

    image = image.crop((left, top, right, bottom))  # defines crop points
    image = image.convert('RGB')
    image.save(path, 'png')  # saves new cropped image

def your_main_method():
    some_element_img = driver.find_element_by_xpath('//*[@id="id-of-image"]')
    save_image_on_disk(driver, some_element_img, 'my-image.png')

About the time you should use time.sleep(6*60)

user_8545280

The image data is located in the src property of the currentSnap element. It's encoded in base64, so you need to capture it and decode it. Then using PIL you can do anything you like with the image.

Also you can use selenium's built in wait functions instead of hardcoding sleeps. In this case the image sometimes loads even after the image element loads, so there's an additional short sleep still in the code to allow it to load.

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


from PIL import Image
from io import BytesIO
import base64
import re

# Max time to wait for page to load
timeout=10

driver = webdriver.Chrome()
driver.get('https://its.txdot.gov/ITS_WEB/FrontEnd/default.html?r=SAT&p=San%20Antonio&t=cctv')

# Wait for element to load before clicking
element_present = EC.presence_of_element_located((By.ID, 'LP-1604'))
WebDriverWait(driver, timeout).until(element_present)

driver.find_element_by_id('LP-1604').click() #to get to the 1604 tab

# Waat for image to load before capturing data
element_present = EC.presence_of_element_located((By.ID, 'currentSnap'))
WebDriverWait(driver, timeout).until(element_present)

# Sometimes the image still loads after the element is present, give it a few more seconds
time.sleep(4)
# Get base64 encoded image data from src
pic = driver.find_element_by_id('currentSnap').get_attribute('src')

# Strip prefix
pic = re.sub('^data:image/.+;base64,', '', pic)

# Load image file to memory
im = Image.open(BytesIO(base64.b64decode(pic)))

# Write to disk
im.save('image.jpg')

# Display image in Jupyter
im
# Open in your default image viewer
im.show()
user_47183500
Related