Intermittent BeautifulSoup with Amazon Book ISBN

user_9061585 0
Viewed 103

I am trying to gather some information about some books available on Amazon and I am having a weird glitch error that I can't understand. At first I thought it was Amazon blocking my connection but then I noticed the request has a "200 OK" and it had the real HTML content of the corresponding page.

Let's take for example this book: https://www.amazon.co.uk/All-Rage-Cara-Hunter/dp/0241985110

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'}

url = 'https://www.amazon.co.uk/All-Rage-Cara-Hunter/dp/0241985110/ref=sr_1_1?crid=2PPCQEJD706VY&dchild=1&keywords=books+bestsellers+2020+paperback&qid=1598132071&sprefix=book%2Caps%2C234&sr=8-1'

response = requests.get(url, headers=headers)

soup = BeautifulSoup(response.content, features="lxml")

price = {}

if soup.select("#buyBoxInner > ul > li > span > .a-text-strike") != []:
    price["regular_price"] = float(
        soup.select("#buyBoxInner > ul > li > span > .a-text-strike")[0].string[1:].replace(",", "."))
    price["promo_price"] = float(soup.select(".offer-price")[0].string[1:].replace(",", "."))
else:
    price["regular_price"] = float(soup.select(".offer-price")[0].string[1:].replace(",", "."))
price["currency"] = soup.select(".offer-price")[0].string[0]

This part works fine and I can have the regular price and a promo price (if exists), and even the currency. But when I do this:

isbn = soup.select("td.bucket > .content > ul > li")[4].contents[1].string.strip().replace("-", "")

I get "IndexError: list index out of range". But if I debug the code, the content is actually there!

Is this a bug of BeautifulSoup? Is the request response too long?

python-requests beautifulsoup python
2 Answers

It seems that Amazon returns two version of the page. One where's <td class="bucket"> and one where are several <span> tags. This script tries to extract ISBN from both of them:

import requests
from bs4 import BeautifulSoup


headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'}

url = 'https://www.amazon.co.uk/All-Rage-Cara-Hunter/dp/0241985110'
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.content, features="lxml")

isbn_10 = soup.select_one('span.a-text-bold:contains("ISBN-10"), b:contains("ISBN-10")').find_parent().text
isbn_13 = soup.select_one('span.a-text-bold:contains("ISBN-13"), b:contains("ISBN-13")').find_parent().text

print(isbn_10.split(':')[-1].strip())
print(isbn_13.split(':')[-1].strip())

Prints:

0241985110
978-0241985113
user_100359850

I wish I had an explanation of the problem but you a solution would be to wrap your code in a function like so:

def scrape():
    headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'}

    url = 'https://www.amazon.co.uk/All-Rage-Cara-Hunter/dp/0241985110/ref=sr_1_1?crid=2PPCQEJD706VY&dchild=1&keywords=books+bestsellers+2020+paperback&qid=1598132071&sprefix=book%2Caps%2C234&sr=8-1'

    response = requests.get(url, headers=headers)

    soup = BeautifulSoup(response.content, features="lxml")

    price = {}

    if soup.select("#buyBoxInner > ul > li > span > .a-text-strike") != []:
        price["regular_price"] = float(
            soup.select("#buyBoxInner > ul > li > span > .a-text-strike")[0].string[1:].replace(",", "."))
        price["promo_price"] = float(soup.select(".offer-price")[0].string[1:].replace(",", "."))
    else:
        price["regular_price"] = float(soup.select(".offer-price")[0].string[1:].replace(",", "."))
    price["currency"] = soup.select(".offer-price")[0].string[0]


    #ADD THIS FEATURE TO YOUR CODE
    isbn = soup.select("td.bucket > .content > ul > li")
    if not isbn:
        scrape()


    isbn = isbn[4].contents[1].string.strip().replace("-", "")

Then if it fails it will just call itself again. You might want to refactor it so it only makes the request once.

user_127597970
Related