Returning Empty list while getting text from span tag (Web scraping)

user_11992075 0
Viewed 460

I am trying to get the product price and name from this URL.

When I inspect the HTML element, the price of the product is under a span tag with the class name c13VH6.

However, I am getting an empty string.

Below is my code.

import requests

from bs4 import BeautifulSoup

headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) 
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'}


html_text=requests.get('https://www.daraz.com.np/televisions/? 
spm=a2a0e.11779170.cate_3.1.287d2d2bmERvcc',headers=headers).text

soup=BeautifulSoup(html_text, 'lxml')
#soup = BeautifulSoup(html_text, 'html5lib')

string1=soup.find('span',class_='c13VH6')
print(soup.find('span', class_='c13VH6'))
print(string1)
web web-scraping beautifulsoup python
1 Answers

The website is loaded dynamically, therefore the requests module won't support it. However, the data is embedded on the website in JSON format, you can find the data using the built-in re (regex) module, and convert it to a Python dictionary (dict) and access the key/value's using the built-in json module.

For example, to print all the products names and prices:

import json
import re
import requests
from bs4 import BeautifulSoup


headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3"
}
url = "https://www.daraz.com.np/televisions/?spm=a2a0e.11779170.cate_3.1.287d2d2bmERvcc"
soup = BeautifulSoup(requests.get(url, headers=headers).content, "lxml")

json_data = json.loads(re.search("window\.pageData=({.*})", str(soup)).group(1))

# To view all the JSON data:
# print(json.dumps(json_data, indent=3))

for data in json_data["mods"]["listItems"]:
    print(data["name"])
    print(data["priceShow"].split()[1])
    print("-" * 80)

Output:

SONY  Klv-32W602D 32 Class Hdr Led Smart Tv - (Black)"
59,500
--------------------------------------------------------------------------------
SONY  Bravia Klv32R302E 32" Hd Led Tv
48,000
--------------------------------------------------------------------------------
Samsung  Ua49N5300Arshe 49 Inch Full Hd Smart Led Tv - Black
87,290
--------------------------------------------------------------------------------
X96 Mini, Android Box, Make Your Any TV Smart & 4K
4,200
--------------------------------------------------------------------------------
Sansui 43S803A 43 Inch Screen 1080P Led Tv
50,369
--------------------------------------------------------------------------------
Samsung  Ua49Nu7100Rshe 49 Inch 4K Ultra Hd Smart Led Tv - Black
128,690
--------------------------------------------------------------------------------
...
...
user_123497340
Related