Scraping the URLs of dynamically changing images from a website

user_13269283 0
Viewed 690

I'm creating a python program that collects images from this website by Google

enter image description here

The images on the website change after a certain number of seconds, and the image url also changes with time. This change is handled by a script on the website. I have no idea how to get the image links from it.

I tried using BeautifulSoup and the requests library to get the image links from the site's html code:

import requests
from bs4 import BeautifulSoup

url = 'https://clients3.google.com/cast/chromecast/home'
html = requests.get(url).text
soup = BeautifulSoup(html, 'html.parser')
tags = soup('img')
for tag in tags:
    print(tag)

But the code returns: 

{{background_url}}' in the image src ("ng-src")

For example:

<img class="S9aygc-AHe6Kc" id="picture-background" image-error-handler="" image-index="0" ng-if="backgroundUrl" ng-src="{{backgroundUrl}}"/>

How can I get the image links from a dynamically changing site? Can BeautifulSoup handle this? If not what library will do the job?

python-requests web-scraping beautifulsoup python
3 Answers
import requests
import re


def main(url):
    r = requests.get(url)
    match = re.search(r"(lh4\.googl.+?mv)", r.text).group(1)
    match = match.replace("\\", "").replace("u003d", "=")
    print(match)


main("https://clients3.google.com/cast/chromecast/home")
user_76589850

Just a minor addition to the answer by αԋɱҽԃ αмєяιcαη (ahmed american) in case anyone is wondering

The subdomain (lhx) in lhx.google.com is also dynamic. As a result, the link can be lh3 or lh4 et cetera.

This code fixes the problem:

import requests
import re


r = requests.get("https://clients3.google.com/cast/chromecast/home").text
match = re.search(r"(lh.\.googl.+?mv)", r).group(1)
match = match.replace('\\', '').replace("u003d", "=")
print(match)

The major difference is that the lh4 in the code by ahmed american has been replaced with "lh." so that all images can be collected no matter the url.

EDIT: This line does not work:

match = match.replace('\\', '').replace("u003d", "=")

Replace with:

match = match.replace("\\", "")
match = match.replace("u003d", "=")
user_132692830

None of the provided answers worked for me. Issues may be related to using an older version of python and/or the source page changing some things around.

Also, this will return all matches instead of only the first match.

Tested in Python 3.9.6.

import requests
import re

url = 'https://clients3.google.com/cast/chromecast/home'
r = requests.get(url)

for match in re.finditer(r"(ccp-lh\..+?mv)", r.text, re.S):
    image_link = 'https://%s' % (match.group(1).replace("\\", "").replace("u003d", "="))
    print(image_link)
user_112140130
Related