How to allow soup to parse different style Amazon webpages

Question

I am testing my code using different style amazon pages and it cannot parse ones in a certain style. It works for 1 type of page but not the page where different options are presented as tabs.

import bs4, requests

header = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.6',}

goodRes = requests.get('https://www.amazon.com/Automate-Boring-Stuff-Python-Programming/dp/1593275994/', headers=header)
goodRes2 = requests.get('https://www.amazon.com/gp/product/1593277954/', headers=header)
badRes = requests.get('https://www.amazon.com/Automate-Boring-Stuff-Python-Programming-ebook/dp/B00WJ049VU/', headers=header)

soup = bs4.BeautifulSoup(goodRes.text, 'lxml') # change to badRes for error
price = soup.select('.offer-price')
name = soup.select('#productTitle')
author = soup.select('.contributorNameID')

print(name[0].text.strip())
print(author[0].text.strip())                   
print(price[0].text.strip())

the first two URL's (goodRes & goodRes2) are direct links to ebooks, they are retrieved by get, and parsed by soup for the information I need correctly.

the third URL is the afformentioned 'tabbed' style page, even though the get request works, soup does not find anything with the given CSS selectors (even though they are exactly the same on this tabbed page). This produces a "list index out of range" error when executing the first print function.

Im not sure how to modify this in order to parse information from the badRes URL.

any help appreciated. Thanks.

user_6241235 · Answer

The same elements, per your selectors, do not exist across all pages. Also, I note that the second link is Kindle and paperback, not ebook, listed; though I guess ebook and kindle are interchangeable. I use Or css syntax to handle different attribute values. For example, for book title you can have either #productTitle, #ebooksProductTitle as ids.

With bs4 4.7.1 you can use :has and :contains pseudo classes to target the price by its relationship to other elements

import bs4, requests

header = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.6',}

urls = ['https://www.amazon.com/Automate-Boring-Stuff-Python-Programming/dp/1593275994/', 'https://www.amazon.com/gp/product/1593277954/', 'https://www.amazon.com/Automate-Boring-Stuff-Python-Programming-ebook/dp/B00WJ049VU/']

with requests.Session() as s:

    for url in urls:
        r = s.get(url, headers=header)
        soup = bs4.BeautifulSoup(r.text, 'lxml') 
        price = soup.select_one('span:has(span:contains("eTextbook"), span:contains("Kindle")) .a-size-base').text.strip()
        author = soup.select_one('.contributorNameID').text
        title = soup.select_one('#productTitle, #ebooksProductTitle').text.strip()
        print(title, author, price)

How to allow soup to parse different style Amazon webpages

1 Answers