Extract images using xpath

user_9950503 0
Viewed 1202

I have been trying to get information from this website https://www.leadhome.co.za/property/poortview-ah/roodepoort/lh-95810/magnificent-masterpiece-in-poortview- and I am having issues with getting all the images of the property; more specifically the URL

this is how the class looks like:

<div class="lazy-image listing-slider-carousel-item lazy-image-loaded">
   <div class="lazy-image-background" style="background-image: url(&quot;https://s3-eu-west-1.amazonaws.com/leadhome-listing-photos/025c90ab-9c87-47d5-b11c-1cfbce3f67f2-md.jpg&quot;);"></div>
</div>

What I have so far:

        for item in response.xpath('//div[@class="lazy-image-background"]/*[starts-with(@style,"background-image")]/@style').getall():
            yield {"image_link":item}

But unfortunately this is empty. Any tips on what I'm doing wrong? Thank you!

scrapy web-scraping xpath html python
3 Answers

If you inspect original html source of this webpage (CTRL + U on google Chrome webbrowser, !!!not html code from Crhome developer tools /elements section)
you will see 2 important things:

  1. Images in tags like <div class="lazy-image listing-slider-carousel-item lazy-image-loaded"> as well as other data don't exists inside these html tags.
  2. All data stored inside script tag and inside window.REDUX_INITIAL_STATE javascript variable:

    original html source

In this case we can convert data from javascript variable into basic python dict format using python's built-in json module.
The most complicated part of this task is to correctly fit content of that script tag into json.loads function. It should be strictly a text after window.REDUX_INITIAL_STATE = and before next javascript operation (in this case before the latest ; symbol). As result we will get this code:

def parse(self, response):
    script_tag = [script for script in response.css("script::text").extract() if "window.REDUX_INITIAL_STATE = {" in script]
    script_data = json.loads(script_tag[0].split("window.REDUX_INITIAL_STATE = ")[-1][:-1], encoding="utf-8")

As you can see on following debugger screenshot all data successfully converted: debugger_converted_data Images stored in script_data['app']['listing']['listing']['entity']['lh-95810']['images'] as list of dictionaries: debugger_images lh-95810 is entity id so in updated code this entity id will be separately selected in order to be able to use it in other pages:

def parse(self, response):
    script_tag = [script for script in response.css("script::text").extract() if "window.REDUX_INITIAL_STATE = {" in script]
    script_data = json.loads(script_tag[0].split("window.REDUX_INITIAL_STATE = ")[-1][:-1], encoding="utf-8")
    entity_key = [k for k in script_data['app']['listing']['listing']['entity'].keys()]
    images = [image["medium"] for image in script_data['app']['listing']['listing']['entity'][entity_key[0]]['images']]

This website uses javascript to render data on webpage. Hovewer any javascript formed content have it's *roots in original html code. This approach uses only built-in json module and don't require css or Xpath selectors.

user_108847910

/* is to look for child element, try without it

response.xpath('//div[@class="lazy-image-background"][starts-with(@style,"background-image")]')
user_51680110

You can use the xpath as:

//div[@class="lazy-image-background" and contains(@style, "background-image: url")]

This will get you all the div elements containing images with url on the page. Then you can add some code to extract image url from each element

user_34658450
Related