Get data-testid and attributes from html using Beautifulsoup

user_7362261 0
Viewed 1681

Web-dev newbie here. so please be nice.

I find this tag really weird for me to parse.

Consider the following HTML doc:

import urllib3
from bs4 import BeautifulSoup

url = 'https://www.carrefourkuwait.com/mafkwt/en/Frozen-Food/c/FKWT6000000?currentPage=1&filter=&nextPageOffset=0&pageSize=60&sortBy=relevance'

req = urllib3.PoolManager()
res = req.request('GET', url)
soup = BeautifulSoup(res.data, 'html.parser')
soup

I am trying to get the product name and price. But using soup.findAll('div', {'data-testid': 'product_name'}) doesn't work.

The issue here is that product name and price are attributes of a link in the <a\> tag. Even with soup.findAll('a') I get nothing: []

enter image description here Can you please help with this?

I also unable to scroll over the pages. I wrote this code but it doesn't work (keep giving me duplicate from page =1)

tag = 'Bakery/c/FKWT1610000' 

scrap_all = pd.DataFrame()

for x in tqdm(range(1,10)):
    scrap_page = pd.DataFrame()
    r = requests.get(parent_url+tag+'?currentPage='+str(x)+'&filter=&nextPageOffset=0&pageSize=200&sortBy=relevance',
                     headers = {'User-Agent':'Mozilla/5.0'})
    
    data = json.loads(re.search(r'(\{"prop.*\})', r.text).group(1))
    data = data['props']['initialState']['search']['products']
    scrap_page['item_desc'] = [i['name'] for i in data]
    scrap_page['item_price'] = [i['originalPrice'] for i in data]

    scrap_carrefour = pd.concat([scrap_carrefour,scrap_page])
web-scraping beautifulsoup html python
1 Answers

Data is dynamically pulled from a script tag. As javascript doesn't run with requests this info remains within the script tag and is not present where you are looking.

You can regex out the string holding the relevant info, parse with json and create a dict as follows:

import requests, re, json

r = requests.get('https://www.carrefourkuwait.com/mafkwt/en/Frozen-Food/c/FKWT6000000?currentPage=1&filter=&nextPageOffset=0&pageSize=60&sortBy=relevance',
                 headers = {'User-Agent':'Mozilla/5.0'})
data = json.loads(re.search(r'(\{"prop.*\})', r.text).group(1))
info = {i['name']:str(i['originalPrice'])+ ' '+ i['currency'] for i in data['props']['initialState']['search']['products']}
user_62412350
Related