Beautifulsoup doesn't work properly with all urls

user_8824494 0
Viewed 48

The error says:

AttributeError: 'NoneType' object has no attribute 'get_text'

I was following a web scraping tutorial and everything worked fine just with this url, when I wanted to change it to this url the error that I already mentioned appeared.

The crawler function:

def product_crawler():
    page = requests.get(url, headers=headers)
    soup = BeautifulSoup(page.content, 'html.parser')
    title = soup.find(id="productTitle").get_text()
    print(title)

I checked all the answers on stackoverflow such as changing html.parser to lxml , but non of them worked.

attributeerror web-scraping beautifulsoup python-3.x
1 Answers

Try to add Accept-Language HTTP header:

import requests
from bs4 import BeautifulSoup

url = "https://www.amazon.com/dp/B08DK5ZH44"

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0",
    "Accept-Language": "en-US,en;q=0.5",
}

page = requests.get(url, headers=headers)
soup = BeautifulSoup(page.content, "html.parser")
title = soup.find(id="productTitle").get_text(strip=True)
print(title)

Prints:

GoPro HERO9 Black - Waterproof Action Camera with Front LCD and Touch Rear Screens, 5K Ultra HD Video, 20MP Photos, 1080p Live Streaming, Webcam, Stabilization
user_100359850
Related