Scraping data using beautiful soup

HRolde 0
Viewed 22

I am fairly new to web scraping / python and have a block of code which I need a little bit of help with. I can't for the life of me see where I've gone wrong with it. Help as always much appreciated. If you have the time, please highlight where I went wrong!

import requests
from bs4 import BeautifulSoup
import csv

page = requests.get("https://www.bathroomspareparts.co.uk/merlyn-two-panel-hinged-bath-screen-mb7-spare-parts-17909-c.asp")
soup = BeautifulSoup(page.content, 'html.parser')


all_products = []


products = soup.select('div.row cf')
for product in products:
    name = product.select('div.item-short-description')[0].text.strip() 
    price = product.select('div.item-price')[0].text.strip() 

    all_products.append({
        "Name": name,
        "Price": price,
    })
web-scraping python
1 Answers

Some products don't have prices so you have to check for that:

import csv
import requests
from bs4 import BeautifulSoup

page = requests.get(
    "https://www.bathroomspareparts.co.uk/merlyn-two-panel-hinged-bath-screen-mb7-spare-parts-17909-c.asp"
)
soup = BeautifulSoup(page.content, "html.parser")

all_products = []

products = soup.select("#products section")
for product in products:
    name = product.select_one(".item-short-description").text.strip()

    price = product.select_one(".item-price")
    price = price.text.strip() if price else "N/A"

    all_products.append(
        {
            "Name": name,
            "Price": price,
        }
    )

print(all_products)

Prints:

[
    {"Name": "Merlyn Bath Screen Cover Caps M7012", "Price": "£1.24"},
    {"Name": "Merlyn Luna Rail End Cap LH SP0M7010/L", "Price": "£1.81"},
    {"Name": "Merlyn Luna Rail End Cap RH SP0M7010/R", "Price": "£1.81"},
    {"Name": "Merlyn Luna Rail SP0M7005", "Price": "£6.54"},
    {
        "Name": "Merlyn Two Panel Hinged Bath Screen MB7 Spare Parts",
        "Price": "N/A",
    },
]
Andrej Kesely0
Related