HREF Class changing on every page

Lalit Joshi 0
Viewed 26

I am working to scrape the website:- "https://www.moglix.com/automotive/car-accessories/216110000?page=101" NOTE: 101 is the page number and this site has 783 pages.

I wrote this code to get all the URL's of the product mentioned on the page using beautifulsoup:-

prod_url = []
for i in range(1,400):
    r = requests.get(f'https://www.moglix.com/automotive/car-accessories/216110000?page={i}')
    soup = BeautifulSoup(r.content,'lxml')
    for link in soup.find_all('a',{"class":"ng-tns-c100-0"}):
        prod_url.append(link.get('href'))

There are 40 products on each page, and this should give me 16000 URLs for the products but I am getting 7600(approx)

After checking I can see that the class for a tag is changing on pages. For Eg:-

The class for a tag on page - 101 is <a class="ng-tns-c105-0"

The class for a tag on page - 1 is <a class="ng-tns-c89-0"

And the actual class on the developer tool page is ng-tns-c85-0 constant on every page

How to get this href for all the products on all the pages.

python-requests-html beautifulsoup python-requests
1 Answers

You can use find_all method and specified attrs to get all a tags also further filter it by using split and startswith method to get exact product link URL's

res=requests.get(f"https://www.moglix.com/automotive/car-accessories/216110000?page={i}")
soup=BeautifulSoup(res.text,"html.parser")
x=soup.find_all("a",attrs={"target":"_blank"})
lst=[i['href'] for i in x if (len(i['href'].split("/"))>2 and i['href'].startswith("/"))]

Output:

['/love4ride-steel-tubeless-tyre-puncture-repair-kit-tyre-air-inflator-with-gauge/mp/msnv5oo7vp8d56',
  '/allextreme-exh4hl2-2-pcs-36w-9000lm-h4-led-headlight-bulb-conversion-kit/mp/msnekpqpm0zw52',
  '/love4ride-2-pcs-35-inch-fog-angel-eye-drl-led-light-set-for-car/mp/msne5n8l6q1ykl',..........]
Bhavya Parikh0
Related