Beautifulsoup scrap very few listings prices instead of all listings prices on a page

userns3m1 0
Viewed 30

I want to scrap data from a real estate website for my education project. I am using beautifulsoup. I write following code. Code works properly but shows very less data.

import requests
from bs4 import BeautifulSoup
url = "https://www.zillow.com/homes/San-Francisco,-CA_rb/"
headers = {
    "Accept-Language": "en-GB,en;q=0.5",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101  Firefox/104.0"
}
response = requests.get(url=url, headers=headers )
soup = BeautifulSoup(response.text, "html.parser")
prices = soup.find_all("span", attrs={"data-test":True})
prices_list = [price.getText().strip("+,/,m,o,1,bd, ") for price in prices]
print(prices_list)

The output of this only shows first 9 listings prices.

['$2,959', '$2,340', '$2,655', '$2,632', '$2,524', '$2,843', '$2,64', '$2,300', '$2,604']

beautifulsoup web-scraping python
1 Answers

It's because the content is created progressively with continuous requests (Lazy loading). You could try to reverse engineer the backend of the site. I'll look into it and if I find an easy solution I'll update the answer. :)

The API call to their backend looks something like this: https://www.zillow.com/search/GetSearchPageState.htm?searchQueryState=%7B%22pagination%22%3A%7B%7D%2C%22usersSearchTerm%22%3A%22San%20Francisco%2C%20CA%22%2C%22mapBounds%22%3A%7B%22west%22%3A-123.07190982226562%2C%22east%22%3A-121.79474917773437%2C%22south%22%3A37.63132659190023%2C%22north%22%3A37.918977518603874%7D%2C%22regionSelection%22%3A%5B%7B%22regionId%22%3A20330%2C%22regionType%22%3A6%7D%5D%2C%22isMapVisible%22%3Atrue%2C%22filterState%22%3A%7B%22sortSelection%22%3A%7B%22value%22%3A%22days%22%7D%2C%22isAllHomes%22%3A%7B%22value%22%3Atrue%7D%7D%2C%22isListVisible%22%3Atrue%7D&wants={%22cat1%22:[%22mapResults%22]}&requestId=3

You need to handle cookies correctly in order to see the results but if delivers around 1000 results. Have fun :)

UPDATE:

should look like this

import json

with open("GetSearchPageState.json", "r") as f:
    a = json.load(f)
    print(a["cat1"]["searchResults"]["mapResults"])
Lost_coder0
Related