How to fix the error "'NoneType' object has no attribute 'text'"

user_16159917 0
Viewed 148

I am trying to scrape some contents on a website. I keep getting this error, 'NoneType' object has no attribute 'text' but I don't know how to fix this.

I noticed that the error has to do with this line tr.find("a", class_="sc-dakcWe sc-liNYZW cPIBpC").text.replace("\n", " ") but I have been stucked on how to fix it. After removing the .text.replace("\n", " ") part from it, I get the response as None. I realise my issue is how to get the correct selector for it, what could I change this tr.find("a", class_="sc-dakcWe sc-liNYZW cPIBpC") to that would give me the correct restaurant_name.

I am using the zomato restaurant and an example url for this is https://www.zomato.com/kanpur/top-restaurants

python code

import requests
from bs4 import BeautifulSoup

city = input("Enter your city: ")
url = "https://www.zomato.com/" + city + "/top-restaurants"
header = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"}
response = requests.get(url, headers=header)

html = response.text
soup = BeautifulSoup(html, "html.parser")
print(soup.title.text)

top_rest = soup.find("div", class_="bke1zw-0 cMipmx")
list_tr = top_rest.find_all("div", class_="bke1zw-1")


for tr in list_tr:
    restaurant_name = tr.find("a", class_="sc-dakcWe sc-liNYZW cPIBpC").text.replace("\n", " ")
    print(restaurant_name)
web-scraping html python
1 Answers

Their classes are dynamically defined it seems so it changes on reload. There seems to be a much more simplistic method (and possibly more reliable) that doesn't require accessing the <a> tag you're trying to do. The images use alt texts for the restaurants, which we can capitalize on:

Code

import requests
from bs4 import BeautifulSoup

city = input("Enter your city: ")
url = "https://www.zomato.com/" + city + "/top-restaurants"
header = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"}
response = requests.get(url, headers=header)

html = response.text
soup = BeautifulSoup(html, "html.parser")
print(soup.title.text)

top_rest = soup.find("div", class_="bke1zw-0 cMipmx")

# NEW METHOD
images = top_rest.find_all("img") # look for all images in top_rest
image_alts = [image.get('alt', '') for image in images] # get their alt texts

print(image_alts)

Result:

Enter your city: kanpur Top Restaurants in Kanpur | Zomato ['Grill Inn', 'Shri Bhojnalaya Restaurant & Sweets', 'Barbeque Nation', 'Kukkad at Nukkad', 'Tadka The Food Hub', 'Smile Pizza', 'Arabian Broost Chicken', 'Chac hi Vaishno Dhaba', 'Barra House', "Pashtun's", 'Agra Vala Sweets', 'Al-Baik.Com' , 'The Imperial Cord', 'Google Fast Food', 'Baba Foods', 'R S Bhojnalaya', 'Kere la Cafe', 'Mama Hotel', 'Gyan Vaishnav', 'New Pizza Yum', 'Offline Cafe', 'The Chocolate Room', 'Mocha']

user_78716850
Related