I have a function that scrapes item and price data from this site
I also require a category value for each item scraped, the parent value is within the text element of <div class="section-title"> and the child item values are within the text element's of <div class="product-name">
The expected output would be:
[{"item": "Deal 1", "price": "7.00", "category": "SPECIAL OFFERS"}
{"item": "Deal 2", "price": "6.00", "category": "SPECIAL OFFERS"}]
HTML sample:
<div class="section-wrapper">
<div class="section-title">SPECIAL OFFERS</div>
<div class="section">
<div class="ac-product-list">
<div class="page-title category-title" id="cat_991">
SPECIAL OFFERS</div>
<ol class="category-products products-list SPECIAL OFFERS">
<li class="item">
<div class="product-name">Deal 1</div>
<p class="description">Chicken Fillet Sandwich, Chips<br/> & A Can Of Soft Drink</p>
<div class="menu-right">
<span class="regular-price" id="product-price-52948_1646624279">
£7.00 </span><button style="display:none" type="button" title="Add to Basket" class="button btn-cart" onclick="setLocation('https://charcoalgrilltotton.co.uk/checkout/cart/add/uenc/aHR0cHM6Ly9jaGFyY29hbGdyaWxsdG90dG9uLmNvLnVrLw,,/product/52948/form_key/nWC3b6ZYcwGKrFdy/?options=cart')">
+ </button></div>
</li>
<li class="item">
<div class="product-name">Deal 2</div>
<p class="description">1/4lb Cheeseburger, Chips<br/> & A Can Of Soft Drink</p>
<div class="menu-right">
<span class="regular-price" id="product-price-52949_791176089">
£6.00 </span><button style="display:none" type="button" title="Add to Basket" class="button btn-cart" onclick="setLocation('https://charcoalgrilltotton.co.uk/checkout/cart/add/uenc/aHR0cHM6Ly9jaGFyY29hbGdyaWxsdG90dG9uLmNvLnVrLw,,/product/52949/form_key/nWC3b6ZYcwGKrFdy/?options=cart')">
+ </button></div>
</li>
<li class="item">
<div class="product-name">Deal 3</div>
<p class="description">5 Breaded Chicken Strips, Chips<br/> & A Can Of Soft Drink</p>
<div class="menu-right">
<span class="regular-price" id="product-price-52950_1618366957">
£6.50 </span><button style="display:none" type="button" title="Add to Basket" class="button btn-cart" onclick="setLocation('https://charcoalgrilltotton.co.uk/checkout/cart/add/uenc/aHR0cHM6Ly9jaGFyY29hbGdyaWxsdG90dG9uLmNvLnVrLw,,/product/52950/form_key/nWC3b6ZYcwGKrFdy/?options=cart')">
+ </button></div>
</li>
<li class="item">
<div class="product-name">Deal 4</div>
<p class="description">Chicken Burger, Chips<br/> & A Can Of Soft Drink</p>
<div class="menu-right">
<span class="regular-price" id="product-price-53124_1465833177">
£6.00 </span><button style="display:none" type="button" title="Add to Basket" class="button btn-cart" onclick="setLocation('https://charcoalgrilltotton.co.uk/checkout/cart/add/uenc/aHR0cHM6Ly9jaGFyY29hbGdyaWxsdG90dG9uLmNvLnVrLw,,/product/53124/form_key/nWC3b6ZYcwGKrFdy/?options=cart')">
+ </button></div>
</li>
<li class="item last">
<div class="product-name">Combi Box</div>
<p class="description">4 Nuggets<br/> 4 Onion Rings<br/> Chips
</p>
<div class="menu-right">
<span class="regular-price" id="product-price-53076_1586614135">
£4.50 </span><button style="display:none" type="button" title="Add to Basket" class="button btn-cart" onclick="setLocation('https://charcoalgrilltotton.co.uk/checkout/cart/add/uenc/aHR0cHM6Ly9jaGFyY29hbGdyaWxsdG90dG9uLmNvLnVrLw,,/product/53076/form_key/nWC3b6ZYcwGKrFdy/')">
+ </button></div>
</li>
My code:
def totton_parse(soup: BeautifulSoup, vendor: str) -> list:
"""Returns list of dict: items, prices, category, vendor"""
items = [i.text.strip() for i in soup.find_all("div", class_="product-name")]
prices = [i.text.strip().lstrip("£") for i in soup.find_all("span", class_="regular-price")]
category = "value" # todo get category for each item
item_stack = []
for it, pr in zip(items, prices):
item_dict = {"vendor": vendor, "items": it, "price": float(pr), "category": category}
item_stack.append(item_dict)
return item_stack
I have tried using find_parent and find_previous with no desired result.
Thanks in advance!