Saving non-repetetive images while crawling website in Selenium

user_16897043 0
Viewed 73

The webpage example is: link

The goal is to download all of the images, but once for each one of them.

Here's the code I'm using:

links = []
wait = WebDriverWait(driver, 5)
all_images = wait.until(
    EC.presence_of_all_elements_located((By.XPATH, "//div[contains(@class,'swiper-button-next swiper-button-white')]")))

for image in all_images:
    a = image.get_attribute('style')
    b = a.split("(")[1].split(")")[0].replace('"', '')
    links.append(b)

all_images = wait.until(
    EC.presence_of_all_elements_located((By.XPATH, "//div[contains(@class,'swiper-slide swiper-slide-visible swiper-slide-active swiper-slide-thumb-active')]")))

for image in all_images:
    a = image.get_attribute('style')
    b = a.split("(")[1].split(")")[0].replace('"', '')
    links.append(b)

all_images = wait.until(
    EC.presence_of_all_elements_located((By.XPATH, "//div[contains(@class,'swiper-slide swiper-slide-visible')]")))

for image in all_images:
    a = image.get_attribute('style')
    b = a.split("(")[1].split(")")[0].replace('"', '')
    links.append(b)

index = 1
for i in range(len(links)//2 + 1):
    with open(title.replace(' ', '-') + str(index) + '.jpg', 'wb') as file:
        im = requests.get(links[i])
        file.write(im.content)
        print('Saving image.. ', title + str(index))
    index += 1

The problem is, it saves repetitive images, and doesn't save some of them, I don't know where I went wrong.

webdriverwait selenium-webdriver selenium web-crawler python
1 Answers

You are using a wrong locator.
Additionally, presence_of_all_elements_located doesn't wait for ALL the elements, it waits for presence of at least 1 element.
Also, presence of element waits for element presence, while this may be not enought. It is recommended to use visibility_of_element_located instead.
I think the following code will work better:

links = []
wait = WebDriverWait(driver, 20)
wait.until(EC.visibility_of_element_located((By.XPATH, "//div[contains(@class,'swiper-slide')]")))
time.sleep(0.5)
all_images = driver.find_elements_by_xpath("//div[contains(@class,'swiper-slide')]")
user_34854340
Related