How to fetch/scrape all elements from a html "class" which is inside "span"?

user_7431372 0
Viewed 71

I am trying to scrape data from a website where i am collecting data from all elements under "class" which is inside "span" using this piece of code. But i am ending up in fetching only one element instead of all.

expand_hits = soup.findAll("a", {"class": "sold-property-listing"})
apartments = []
for hit_property in expand_hits:
    #element = soup.findAll("div", {"class": "sold-property-listing__location"})
    place_name = expand_hits[1].find("div", {"class": "sold-property-listing__location"}).findAll("span", {"class": "item-link"})[1].getText()
    print(place_name)
    apartments.append(final_str)

Expected result for print(place_name)

Stockholm
Malmö
Copenhagen
...
..
.

The result which is am getting for print(place_name)

Malmö
Malmö
Malmö
...
..
.

When i try to fetch the contents from expand_hits[1] i get only one element. If i don't specify the index scraper is throwing an error regarding the usage find(), find_all() and findAll(). As far as i understood i think i have to call the content of the elements iteratively.

Any help is much appreciated. Thanks in Advance!

data-science beautifulsoup python
1 Answers

Use the loop variable rather than indexing to same collection with same index (expand_hits[1]) and append place_name not final_str

expand_hits = soup.findAll("a", {"class": "sold-property-listing"})
apartments = []
for hit_property in expand_hits:
    place_name = hit_property.find("div", {"class": "sold-property-listing__location"}).find("span", {"class": "item-link"}).getText()
    print(place_name)
    apartments.append(place_name)

You only then need Find and no indexing

Add User-Agent header to ensure results. Also, I note that I have to pick a parent node because at least one result will not be captured by using that class item-link e.g. Övägen 6C. I use replace to get rid of the hidden text present due to now selecting for parent node.

from bs4 import BeautifulSoup 
import requests
import re

url = "https://www.hemnet.se/salda/bostader?location_ids%5B%5D=474035"
page = requests.get(url, headers = {'User-Agent':'Mozilla/5.0'})
soup = BeautifulSoup(page.content,'html.parser')

for result in soup.select('.sold-results__normal-hit'):
    print(re.sub(r'\s{2,}',' ', result.select_one('.sold-property-listing__location h2 + div').text).replace(result.select_one('.hide-element').text.strip(), ''))

If you only want where in Malmo e.g. Limhamns Sjöstad, you need to check how many child span tags there are for each listing.

for result in soup.select('.sold-results__normal-hit'):
    nodes = result.select('.sold-property-listing__location h2 + div span')
    if len(nodes)==2:
        place = nodes[1].text.strip()
    else:
        place = 'not specified'    
    print(place)
user_62412350
Related