How to scrape different values that share the same class name in Python & Beautifulsoup?

Viewed 24

I am trying to scrape a real estate website but I have ran into a div with same class name(class="number") but has 3 different values("bedroom", "bathroom" & "garage". Here's the data I'm trying to scrape:

<div class="infoHolder">
        
        <div class="title">3 Bed Townhouse in Ferndale</div>
        <div class="priceDescription">R 1 280 000</div>
        <div class="priceAdditionalDescriptor"></div>

        
            <div class="propertyType">Townhouse</div>
                    <div class="suburb">Ferndale</div>
                <div class="features row">
                <div class="number">3</div>
                <div class="icon bedroom"></div>                
                            <div class="number">2</div>
                <div class="icon bathroom"></div>
                            <div class="number">1</div>
                <div class="icon garage"></div>

        </div>

Here's the code I'm using but it doesn't yield expected results:

import requests
from bs4 import BeautifulSoup as bs

page = 1
prices = []
property_type = []
locations = []
str_address= []
bedrooms = []
bathrooms = []
parking_space = []
while page != 57:
    url = f"https://www.privateproperty.co.za/for-sale/gauteng/johannesburg/randburg-and-ferndale/35?page={page}"
    response = requests.get(url)
    soup = bs(response.content, "html.parser")
    for price in soup.findAll('div', class_="priceDescription"):
        prices.append(price.get_text(strip=True))
    for p_type in soup.findAll('div', class_="propertyType"):
        property_type.append(p_type.get_text(strip=True))
    for location in soup.findAll("div", class_="suburb"):
        locations.append(location.get_text(strip=True))
    for street_address in soup.findAll('div', class_="address"):
        str_address.append(street_address.get_text(strip=True))
    for el in soup.findAll("div", class_="number"):
        bedrooms.append(el.get_text(strip=True))
    for br in soup.findAll("div", class_="number"):
        bathrooms.append(br.get_text(strip=True))
    for parking in soup.findAll("div", class_="number"):
        parking_space.append(parking.get_text(strip=True))
    page = page + 1

I'd really appreciate your help. Thanks in advance.

1 Answers

You can use findNextSibling to find a tag after a specified tag. Eg. first search for the div with class "icon bedroom", and use findNextSibling to find the div with class "number" thereafter:

from bs4 import BeautifulSoup as bs

html = '''<div class="infoHolder">
        
        <div class="title">3 Bed Townhouse in Ferndale</div>
        <div class="priceDescription">R 1 280 000</div>
        <div class="priceAdditionalDescriptor"></div>

        
            <div class="propertyType">Townhouse</div>
                    <div class="suburb">Ferndale</div>
                <div class="features row">
                <div class="number">3</div>
                <div class="icon bedroom"></div>                
                            <div class="number">2</div>
                <div class="icon bathroom"></div>
                            <div class="number">1</div>
                <div class="icon garage"></div>

        </div>'''

soup = bs(html, "html.parser")

bedrooms = soup.find('div', class_="icon bedroom").findNextSibling('div', class_="number").get_text()

Output: 2

Related