looking for item in web scrape

Admin 1
Viewed 45

I was looking for a way to scrape the author and price in Amazon store. (Then remove the dollar sign, leaving it like 3.99 in the output.)

I had tried and gain the title and rating so far, but not sure how can I retrieve the name of the author.

from bs4 import BeautifulSoup
import pandas as pd
import requests

url = '             '
html_content = requests.get(url).text
soup = BeautifulSoup(html_content, "html")

rate = []

for x in soup.select("li.zg-item-immersion"):
    item = {}
    
    item['name'] = x.select_one('a').get_text(strip=True)

    item['rank'] = x.select_one('span span').get_text(strip=True)

    rate.append(item)
        
rate

In the output after the above input, I get:

enter image description here

just wonder how can I remove the items in the brackets for every name section.

So that for example 'Right Behind Her (Bree Taggert Book 4)'

can be 'Right Behind Her'

web-scraping beautifulsoup python
1 Answers

You can use this code :

x.find("span", {"class": "p13n-sc-price"}).get_text().split('$')[1]

Whole code:

from bs4 import BeautifulSoup
import pandas as pd
import requests

url = 'https://www.amazon.com/Best-Sellers-Kindle-Store/zgbs/digital-text'
html_content = requests.get(url).text
soup = BeautifulSoup(html_content, "html")

rate = []

for x in soup.select("li.zg-item-immersion"):
    item = {}
    
    item['name'] = x.select_one('a').get_text(strip=True).split('(')[0].strip()

    item['rank'] = x.select_one('span span').get_text(strip=True)
    
    item['price'] = x.find("span", {"class": "p13n-sc-price"}).get_text().split('$')[1]
    try:
        item['author'] = x.find("a", {"class":"a-size-small a-link-child"}).text
    except:
        item['author'] = 'Not Found Author Name'

    rate.append(item)

Output:

[{'name': 'Peril', 'rank': '#1', 'price': '14.99', 'author': 'Bob Woodward'},
 {'name': 'Apples Never Fall',
  'rank': '#2',
  'price': '14.99',
  'author': 'Liane Moriarty'},
...
]
user_17405770
Related