Beautifulsoup for webscraping is not working?

user_6283073 0
Viewed 510

I'm trying to scrape some data from a website. This is the html format. I want to scrape the word "No description for 632930413867".

Html Code:

<div class="col-xs-6 col-sm-6 col-md-6 col-lg-6">
  <table class="table product_info_table">
    <tbody>
      <tr>
        <td>GS1 Address</td>
        <td>R.R. 1, Box 2, Malmo, NE 68040</td>
      </tr>
      <tr>
        <td>Description</td>
        <td>
          <div id="read_desc">
            No description for 632930413867
          </div>
        </td>
      </tr>
    </tbody>
  </table>
</div>

and the image src from this html

  <div class="centered_image header_image">
<img src="https://images-na.ssl-images-amazon.com/images/I/416EuOE5kIL._SL160_.jpg" title="UPC 632930413867" alt="UPC 632930413867">

so I use this code 

Baseurl = "https://www.buycott.com/upc/632930413867"
uClient = ''
while uClient == '':
    try:
        uClient = requests.get(Baseurl)
        print("Relax we are getting the data...")

    except:
        print("Connection refused by the server..")
        print("Let me sleep for 7 seconds")
        time.sleep(7)
        print("Was a nice sleep, now let me continue...")
        continue


page_html = uClient.content

uClient.close()
page_soup = soup(page_html, "html.parser")

Productcontainer = page_soup.find_all("div", {"class": "row"})
link = page_soup.find(itemprop="image")

print(Productcontainer)

for item in Productcontainer:
    print(link)
    productdescription = Productcontainer.find("div", {"class": "product_info_table"})
    print(productdescription)

when I run this code no data is displayed. How can I obtain the description and img src?

beautifulsoup html python
3 Answers

There is just one instance of each (item and productdescription) on the page so you can go to them directly using find(), there is no need to use find_all() in this case:

import requests
from bs4 import BeautifulSoup as soup

Baseurl = "https://www.buycott.com/upc/632930413867"
uClient = ''
while uClient == '':
    try:
        uClient = requests.get(Baseurl)
        print("Relax we are getting the data...")

    except:
        print("Connection refused by the server..")
        print("Let me sleep for 7 seconds")
        time.sleep(7)
        print("Was a nice sleep, now let me continue...")
        continue

page_html = uClient.content
uClient.close()

page_soup = soup(page_html, "html.parser")
productdescription = page_soup.find("div", {"id": "read_desc"}).text
link = page_soup.find("div", {"class": "centered_image header_image"}).find("img")['src']
print (productdescription)
print (link)

Outputs:

Relax we are getting the data...

No description for 632930413867

https://images-na.ssl-images-amazon.com/images/I/416EuOE5kIL._SL160_.jpg
user_8393380

You just have to inspect the html and identify the tags that hold the data you want to scrape.
In this case it's div.centered_image.header_image img for the image and div#read_desc for the description.
An example with bs4 css selectors: 

import requests
from bs4 import BeautifulSoup 

baseurl = "https://www.buycott.com/upc/632930413867"
page_html = requests.get(baseurl).content
soup = BeautifulSoup(page_html, "html.parser")
image = soup.select_one('div.centered_image.header_image img')['src']
description = soup.select_one('div#read_desc').text.strip()

print(image)
print(description)

https://images-na.ssl-images-amazon.com/images/I/416EuOE5kIL.SL160.jpg
No description for 632930413867

user_78116730

This can be done like this as well:

import requests
from bs4 import BeautifulSoup

soup = BeautifulSoup(requests.get("https://www.buycott.com/upc/632930413867").text, "lxml")
desc = soup.select("#read_desc")[0].text.strip()
link = soup.select(".centered_image img")[0]['src'].strip()
print("{}\n{}".format(desc,link))

Output:

No description for 632930413867
https://images-na.ssl-images-amazon.com/images/I/416EuOE5kIL._SL160_.jpg
user_91897990
Related