I can't get the text from an URL (BeautifullSoup)

user_12927799 0
Viewed 50

I'm trying to learn how to get info from the internet by a python script. Here it is:

import requests
from bs4 import BeautifulSoup

url = 'https://www.amazon.es/Poco-NFC-Smartphone-Punch-Hole-Quad-c%C3%A1mara/dp/B08B9C149J/ref=sr_1_3?__mk_es_ES=%C3%85M%C3%85%C5%BD%C3%95%C3%91&dchild=1&keywords=poco%2Bx3&qid=1600179415&quartzVehicle=16-179&replacementKeywords=poco&sr=8-3&th=1'


headers ={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36'}


page = requests.get(url, headers=headers)

soup = BeautifulSoup(page.content, "html.parser")

title = soup.find( id='productTitle').get_text()

The problem is that I'm getting none as a result of the ID 'productTitle' search when I should get the title I'm seing in the browser.

Any ideas?

beautifulsoup python
3 Answers

When going to the link provided in the code and then searching manually for "productTitle" in the HTML code, three results appear. The issue might be that you simply get the first result and therefor not the actual title. I am guesing you want the span with the id=productTitle. Try using

soup.find("span", id='productTitle').get_text()
user_142812230

The problem here is that the element with id="productTitle" doesn't exist in the content in the response. You can verify this by opening the Sources tab in Chrome or another browser. Press Ctrl+Shift+I and click on Sources at the top of the page. Expand www.amazon.es and the folder under it. Click on the file and press Ctrl+F and type "productTitle". It is not found.

You will also see that this page has a lot of JavaScript. Most likely when that code runs in the browser it adds elements to the DOM. When you use requests and BeautifulSoup, this JavaScript code is not executed so you won't get the exact same results as what you see in the browser.

user_14405650
user_141669520
Related