how to return date and title from the below link using BeautifulSoup in python

user_5980666 0
Viewed 79

I want to extract data from the link below using BeautifulSoup package in python I am trying to get all the links of the first page and then get all the related data of each link

example as : publish_date & title

but the system crash and display the below error :

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-95-0fd35627bc48> in <module>
     52     s = BeautifulSoup(requests.get(link).content, "lxml")
     53 
---> 54     date_published = s.find("span", class_="t-mute").getText(strip=True)
     55     title = s.find("h1", class_="h3 t-break").getText(strip=True)
     56     print(f"{date_published} {title}\n\n", "-" * 80)

AttributeError: 'NoneType' object has no attribute 'getText'

==================================

import time
import requests
from bs4 import BeautifulSoup

soup = BeautifulSoup(
    requests.get("https://www.bayt.com/en/international/jobs/executive-chef-jobs/").content,
    "lxml"
)


links = []
for a in soup.select("h2.m0.t-regular a"):
    if a['href'] not in links:
        links.append("https://www.bayt.com"+ a['href'])


for link in links:
    
    s = BeautifulSoup(requests.get(link).content, "lxml")

    date_published = s.find("span", class_="t-mute").getText(strip=True)
    title = s.find("h1", class_="h3 t-break").getText(strip=True)
    print(f"{date_published} {title}\n\n", "-" * 80)
web-crawler beautifulsoup python
2 Answers

you search the wrong element


import time
import requests
from bs4 import BeautifulSoup

soup = BeautifulSoup(
    requests.get("https://www.bayt.com/en/international/jobs/executive-chef-jobs/").content,
    "lxml"
)


links = []
for a in soup.select("h2.m0.t-regular a"):
    if a['href'] not in links:
        links.append("https://www.bayt.com"+ a['href'])


for link in links:
    s = BeautifulSoup(requests.get(link).content, "lxml")

    date_info = s.find_all("li", class_="t-mute")[-1]
    date_published = date_info.find("span", class_="u-none").getText(strip=True)

    title = s.find("h1", class_="h3 t-break").getText(strip=True)
    print(f"{date_published} {title}\n\n", "-" * 80)
user_101312420

Read the error message:

 AttributeError: 'NoneType' object has no attribute 'getText'

means that s.find("span", class_="t-mute") is None, i.e. no result was found.

Said otherwise: page structure/tags are not as expected.

So either:

  • fix your search criteria
  • or test for None-ity after your search, before calling getText method.
user_7361510
Related