How to extract beer names with XPATH in Python from this hungarian webshop?

user_14714722 0
Viewed 71

I am trying to extract product information of beers from a hungarian webshop Spar. My code only seems to work and extract the product names when I don't filter to beers. So in case page_url = ('https://www.spar.hu/onlineshop/') then the code works fine but when trying to use the link below in my code, it extracts nothing. I have also tried using the link for total alcohol category but same as when using the beers one.

The code itself:

import requests
from scrapy import Selector


headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36'
}

page_url = ('https://www.spar.hu/onlineshop/alkoholos-italok/sor-cider/c/H8-6/?page=1')

html = requests.get(url = page_url, headers = headers).content

sel = Selector(text = html)

for card in sel.xpath('//div[@class= "productBox"]'):
    name = card.xpath('//label[@class="productTitle"]/a/text()').extract()
    print(name)
web-scraping xpath python
2 Answers

When you are sending a GET request, the URL you mentioned above will only return it's source code.

The source code doesn't include the products, but only a Javascript which will then load the products from a different URL.

If you open up your URL in a Webbrowser and press CTRL+U you will see the source code. If you drill into the page by opening the developer tools in your Webbrowser (F12) and reload the page, you can actually see all the content that the page loads.

When using the network inspection in the developer tools you will come accross this URL: https://sp1004e537.guided.lon5.atomz.com/?page=1&category=H8-6&callback=parseResponse&sp_cs=UTF-8&sort=product-ecr-sortlev&m_sortProdResults_egisp=a&pos=6108207&callback=parseResponse&_=1606416848605

This is the actual URL which returns the list of products as a Json file.

Here is the complete Code for this page:

import requests
import json

headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36'
}

page_url = ('https://sp1004e537.guided.lon5.atomz.com/?page=1&category=H8-6&sp_cs=UTF-8&sort=product-ecr-sortlev&m_sortProdResults_egisp=a&pos=6108207&_=1606416848605')

r = requests.get(url = page_url, headers = headers)

if r.status_code == 200:
    j = json.loads(r.text[14:-1]) # remove non Json string from the beginning and end parseResponse( [....] ) 
    for beer in j['results']:
        print(beer['title-hu'])

I highly recommend to debug the dictionary "beer" since it includes a lot more information.

user_73159670

You could just add a . just before you second xpath, use get() instead of extract() and do a little bit of trimming on the result string.

With the following code :


for card in sel.xpath('//div[@class= "productBox"]'):
    name = card.xpath('.//label[@class="productTitle"]/a/text()').get()
    print(name)

i get this result: Tchibo Cafissimo Special őrölt, pörkölt kávékapszula 6 x 10 db 471 g

user_143333790
Related