How to scrape Iframe from JavaScript website using Scrapy Splash?

Viewed 22

I've installed scrapy-splash to scrape the listings of a JavaScript website called Boston Realty Advisors. I used scrapy shell to ensure everything is running okay. This is the command I used to scrape the website using scrapy-splash:

scrapy shell "http://localhost:8050/render.html?url=https://bradvisors.com/listings/

I then run the following command to access the data I want to scrape from the iframe:

# This returns the iframe URL 
iframe_url = response.xpath("//iframe/@src").get()

# Using splash to scrape the iframe
fetch("http://localhost:8050/render.html?url=" + iframe_url)

This returned a 200 response. To ensure I've accessed the page, I view(response) however, the page fails to load completely - see the image below. Page response when viewed from web scraper

When I search the iframe URL directly from my browser, I receive a completely populated page - See below.

iFrame response when searched directly from browser.

How could I make my web scraper return the populated page so I can scrape the elements? (Ideally, I'd like to stick to scrapy splash).

0 Answers
Related