Scraping website-elements that are using same keys and classes & challenges with jumping in loop using Python's BeautifulSoup
I have been looking everywhere for a solution, talked with a software engineer and my python-professor regarding this, which couldn’t help. This is my first post so please bear with me:
I try to scrape a website using BeautifulSoup to call the elements I want to extract. The main challenge is that the website I try to scrape from, are using the same classes and keys in multiple occations with different elements (In this example: tag = span classes = text-nowrap).
This means my output from the functions defining the elements I want are printing the same (first HTML-line, which has a tag=span and class = text-nowrap from above).
In this instance, I thought I might need an additional parameter in the list.find helping it distinguish as some elements consists of special elements like date-format, m², DKK/m² and so forth? Or is there any other ways?
Furthermore, I seem to have a problem regarding jumping when creating the loop. Printing the soup.find_all class-parameter (in this case, “Container mb-5”), I get all the information I want (unfiltered).However, as soon as I try to create a loop defining the functions of the elements I want printed, the loop only loops once (from the top) giving me one example instead of multiple lines of data. Syntax seems correct.
Below is my Python-code:
pip install beautifulsoup4
pip install lxml
pip install requests
from bs4
import BeautifulSoup
from lxml
import etree
import requests
from csv
import writer
url = "https://www.boliga.dk/salg/resultater?searchTab=1&sort=date-d&saleType=1&propertyType=1,3&salesDateMin=2015"
page = requests.get(url)
print(page)
soup = BeautifulSoup(page.content, "html.parser")
lists = soup.find_all("div", class_ = "container mb-5")
for list in lists:
resitype = list.find('span', class_ = "text").text
address = list.find('a', class_ = "text-primary font-weight-bolder text-left").text
price = list.find('span', class_ = "text-nowrap").text.replace('\xa0kr.', "")
salesdate = list.find('span', class_ = "text-nowrap")
rooms = list.find('td', class_ = "table-col d-print-table-cell text-center")
sqm = list.find('span', class_ = "text-nowrap")
sqmprice = list.find('span', class_ = "text-nowrap mt-1").text
salestype = list.find('span', class_ = "text-nowrap mt-1").text
buildingyear = list.find('span __ngcontent-boliga-app-c182', class_ = "table-col d-print-table-cell text-center")
procent = list.find('a', class_ = "price-reduced ng-star-inserted")
info = [resitype, address, price, salesdate, rooms, sqm, sqmprice, salestype, buildingyear, procent]
print(info)