Web-scraping: Accessing text information within a large list

user_10792871 0
Viewed 41

Example: https://www.realtor.com/realestateandhomes-detail/20013-Hazeltine-Pl_Ashburn_VA_20147_M65748-31771

I am trying to access the number of garage spaces for several real estate listings. The only problem is that the location of the number of garage spaces isn't always in the 9th location of the list. On some pages it is earlier, and on other pages it is later.

garage = info[9].strip().replace('\n','')[15]

where

info = soup.find_all('ul', {'class': "list-default"})
info = [t.text for t in info]

and

header = {"user agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.2 Safari/605.1.15"}
page = requests.get(url, headers = header)
page.reason
requests.utils.default_user_agent()
    
soup = bs4.BeautifulSoup(page.text, 'html5lib')

What is the best way for me to obtain how many garage spaces a house listing has?enter image description here

python-requests web-scraping beautifulsoup python-3.x python
1 Answers

You can use CSS selector li:contains("Garage Spaces:") that will find <li> tag with the text "Garage Spaces:".

For example:

import requests
from bs4 import BeautifulSoup


url = 'https://www.realtor.com/realestateandhomes-detail/20013-Hazeltine-Pl_Ashburn_VA_20147_M65748-31771'
headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.2 Safari/605.1.15"}
soup = BeautifulSoup(requests.get(url, headers=headers).content, 'html.parser')

garage_spaces = soup.select_one('li:contains("Garage Spaces:")')
if garage_spaces:
    garage_spaces = garage_spaces.text.split()[-1]
    print('Found Garage spaces! num =', garage_spaces)

Prints:

Found Garage spaces! num = 2
user_100359850
Related