How to scrape website while iterate on multiple pages

Question

Trying to scrape this website using python beautifulsoup: https://www.leandjaya.com/katalog

having some challenges in navigating the multiple pages of the website and scrape it using python, this website has 11 pages, and curious to know the best option to achieve this like use for loop and will break the loop if the page doesnt exist.

this is my initial code, I have set a big number 50, however seems this is not a good option.
page = 1
while page != 50:
    url=f"https://www.leandjaya.com/katalog/ss/1/{page}/"
    main = requests.get(url)
    pmain = BeautifulSoup(main.text,'lxml')
    page = page + 1
Sample output:
https://www.leandjaya.com/katalog/ss/1/1/
https://www.leandjaya.com/katalog/ss/1/2/
https://www.leandjaya.com/katalog/ss/1/3/
https://www.leandjaya.com/katalog/ss/1/<49>/

Barry the Platipus · Answer

This is one way to extract that info and display it in a dataframe, based on an unknown number of pages with data:

import requests
import pandas as pd
from bs4 import BeautifulSoup as bs

cars_list = []
headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36'
}
s = requests.Session()
s.headers.update(headers)
counter = 1
while True:
    try:
        print('page:', counter)
        url = f'https://www.leandjaya.com/katalog/ss/1/{counter}/'
        r = s.get(url)
        soup = bs(r.text, 'html.parser')
        cars_cards = soup.select('div.item')
        if len(cars_cards) < 1:
            print('all done, no cars left')
            break
        for car in cars_cards:
            car_name = car.select_one('div.item-title').get_text(strip=True)
            car_price = car.select_one('div.item-price').get_text(strip=True)
            cars_list.append((car_name, car_price))
        counter = counter + 1
    except Exception as e:
        print('all done')
        break
df = pd.DataFrame(cars_list, columns = ['Car', 'Price'])
print(df)

Result:

page: 1
page: 2
page: 3
page: 4
page: 5
page: 6
page: 7
page: 8
page: 9
page: 10
page: 11
page: 12
all done, no cars left
Car Price
0   HONDA CRV 4X2 2.0 AT 2001   DP20jt
1   DUJUAL XPANDER 1.5 GLS 2018 MANUAL  DP53jt
2   NISSAN JUKE 1.5 CVT 2011 MATIC  DP33jt
3   Mitsubishi Xpander 1.5 Exceed Manual 2018   DP50jt
4   BMW X1 2.0 AT SDRIVE 2011   DP55jt
... ... ...
146 Daihatsu Sigra 1.2 R AT DP130jt
147 Daihatsu Xenia Xi 2010  DP85jt
148 Suzuki Mega Carry Pick Up 1.5   DP90jt
149 Honda Mobilio Tipe E Prestige   DP150jt
150 Honda Freed Tipe S  Rp. 170jtRp. 165jt
151 rows × 2 columns

The relevant documentations for the packages used above can be found at:

https://beautiful-soup-4.readthedocs.io/en/latest/index.html

https://requests.readthedocs.io/en/latest/

https://pandas.pydata.org/pandas-docs/stable/index.html

How to scrape website while iterate on multiple pages

1 Answers