Beautiful Soup findAll doesn't find value

Question

I want to write the prices of the products on this site with the code below with beautifulsoup, but when I write the code, the list returns empty.

import requests as req from bs4 import BeautifulSoup as bs

url = "https://www.migros.com.tr/temel-gida-c-2?sayfa=1"

headers = { 'User-Agent': ( 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) ' 'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36' ) }

urunler = []

rx = req.get(url, headers=headers) sympo = bs(rx.text, 'html.parser') soup = bs(rx.content, 'lxml') print(sympo.findAll('span', {'class': 'amount'}))

print(sympo.findAll('span', {'class': 'amount'}))
[]

user_9501435 · Answer

The page is generated by a JS call, so you're probably going to have to look into the selenium package. If you just check the output when you print soup, you'll only get the header content for the page:





Migros Sanal Market: Online Market Alışverişi

user_10035985 · Answer

The data you see is loaded via Javascript. You can use requests/json module to load it:

import json
import requests


url = "https://www.migros.com.tr/rest/search/screens/temel-gida-c-2?sayfa=1"
data = requests.get(url).json()

# uncomment this to print all data:
# print(json.dumps(data, indent=4))

for p in data["data"]["searchInfo"]["storeProductInfos"]:
    print(
        "{:<35} {:<10} {:<10}".format(
            p["name"], p["regularPrice"], p["salePrice"]
        )
    )

Prints:

Maydanoz Adet                       445        445       
Soğan Kuru Dökme Kg                 195        195       
Migros Havuç Beypazarı Paket Kg     875        750       
Domates Kg                          1495       1495      
Kabak Sakız Kg                      1990       1990      
Dereotu Adet                        930        930       
Roka Demet                          603        603       
Salata Kıvırcık Adet                1090       1090      
Patlıcan Kemer Kg                   1990       1990      
Soğan Taze Demet                    925        925       
Hıyar Kg                            1790       1790      
Domates Salkım Kg                   2290       2290      
Biber Kırmızı Kg                    2190       2190      
Brokoli Kg                          3450       3450      
Atom Salata Adet                    1206       1206      
Kereviz Kg                          875        875       
Karnabahar Kg                       1390       1390      
Ispanak Kg                          1450       1450      
Patates Taze Kg                     556        556       
Biber Köy Usulü Kg                  2990       2990      
Nane Adet                           631        631       
Biber Sivri Kg                      2690       2690      
Pırasa Kg                           930        930       
Lahana Beyaz Kg                     595        595       
Biber Dolmalık Kg                   2702       2702      
Domates Şeker 250 G                 837        837       
Lahana Kırmızı Kg                   1206       1206      
Patates Ekonomik Boy File Kg        445        445       
Pancar Kg                           743        743       
Domates Pembe Kg                    1850       1850

Beautiful Soup findAll doesn't find value

2 Answers