beautifulsoup4 find_all not finding any data on Walmart grocery website

user_8069134 0
Viewed 265

I am attempting to scrape some basic product information from the url linked here, but the bs4 find_all command isn't finding any data given the name of the class associated with the product div. Specifically, I am trying:

url = https://www.walmart.com/grocery/browse/Cereal-&-Breakfast-Food?aisle=1255027787111_1255027787501
r = requests.get(url)
soup = BeautifulSoup(r.content, 'lxml')
product_list = soup.find_all('div', class_='productListTile')
print(product_list)

But this prints an empty list []. Having inspected the webpage on Chrome, I know that 'productListTile' is the correct class name. Any idea what I am doing wrong?

python-requests web-scraping beautifulsoup python
2 Answers

You will need to use Selenium most likely. Beautiful Soup requests get redirected to a "Verify Your Identity" page.

Here is a very similar question to this one, which has code with Selenium and Beautiful Soup working in concert to scrape Wal-Mart

python web scraping using beautiful soup is not working

user_43523170

Web scraping technics vary with websites. In this case, you can either use selenium that is a good option and here I am adding another method with the beautiful soup itself, this helped me a lot.

In this case, inspect the web page and then select network, please refresh the page. enter image description here

Then sort with type: In the below image I had marked with red color the API's they called to get the data from the backend. So you can directly call the backend API to fetch the player's data. enter image description here

Check the "Headers" you will see the API endpoint and in preview, you can see the API response in JSON format. enter image description here

Now if you want to get the images then please check the source you will see the images and u can download the images and map with the id. enter image description here

user_66847510
Related