Web Scraping in Python container not found

user_14249298 0
Viewed 103

I am trying to scrape data from a web page but I am not able to find all the chunks of data inside a container.

<div class="product-list-container" data-project-b="" data-mini-basket="on" data-chunk-caching-gutter="5" data-chunk-caching-delay="2" data-total-results="112" data-chunks-per-page="8" data-results-per-chunk="24" data-current-chunk="1" data-page-end-chunk="5" data-offset-chunk-threshold="12" data-findability-grid-position="on">

Inside the above container I have five chunks of data like the examples listed below:

<div class="product-list-container__chunk product-list-container__chunk--1" data-chunk="1" data-tagg-processed="true">

<div class="product-list-container__chunk product-list-container__chunk--2" data-chunk="2" style="" data-tagg-processed="true">

I am able to access first container with:

from bs4 import BeautifulSoup
import requests

soup = BeautifulSoup(page.content, 'html.parser')
d = soup.find('div', attrs={'product-list-container__chunk product-list-container__chunk--1'})

But if I try to do the same with the second container I am not able to access it, it seems like the parser is not getting all the information from the website or maybe I am not looking for it correctly.

from bs4 import BeautifulSoup
import requests

soup = BeautifulSoup(page.content, 'html.parser')
d = soup.find('div', attrs={'product-list-container__chunk product-list-container__chunk--2'})
web-scraping python
1 Answers

The page loads the product dynamically as you scroll down. You can use this script to get information about all products:

import requests
from bs4 import BeautifulSoup


url = 'https://www.johnlewis.com/browse/electricals/mobile-phones-accessories/view-all-mobile-phones/_/N-a8v?incremental=true&page={page}&country=uk&currency=gbp'

page = 1
while True:
    soup = BeautifulSoup(requests.get(url.format(page=page)).content, 'html.parser')

    titles = soup.select('.product-card__title')

    if not titles:
        break

    # print some information on screen:
    for t in titles:
        print(t.get_text(strip=True))
        print(t.find_next(class_='product-card__price').get_text(strip=True))
        print('-' * 80)

    page += 1

Prints:

Apple iPhone 11, iOS, 6.1", 4G LTE, SIM Free, 64GB
£679.00
--------------------------------------------------------------------------------
Apple iPhone 11 Pro Max, iOS, 6.5", 4G LTE, SIM Free, 256GB
£1,174.00
--------------------------------------------------------------------------------
Apple iPhone 11 Pro, iOS, 5.8", 4G LTE, SIM Free, 256GB
£1,074.00
--------------------------------------------------------------------------------
Apple iPhone 11 Pro Max, iOS, 6.5", 4G LTE, SIM Free, 64GB
£1,024.00
--------------------------------------------------------------------------------

...and so on.
user_100359850
Related