Hello I am trying to replace url links from string with that url's title tag. Here is the function:
import requests
import urllib.request
import re
from bs4 import BeautifulSoup as bs
import string
def get_title_from_url(url):
#check if it's a word of https string
if(url.startswith('http')):
#check if url is valid
try:
page = requests.get(url)
except:
return ''
#if the url is valid, sometimes phisy/ssl-missing site throw error
try:
html = bs(page.content, 'html.parser')
except:
return ''
#finally get the title if everyting is good
page_title = html.find('title').text.strip()
return page_title
else:
return url
then here is a sample string that gets me the error even though the URL for tweet exists and leads to the AJ news.
s = 'A new virus spreading in China has been detected in Seattle, ' \
'says the CDC — making it the 1st U.S. case.\r\n\r\n▪️ Health officials ' \
'confirmed the virus is contagious\r\n▪️ It has infected ~300 people, and killed ' \
'6, in China\r\n▪️ Cases also in Japan, Thailand, Taiwan, ' \
'South Korea https://twitter.com' #original link to tweet removed due to SO guideline
l = list(map(lambda x: x, s.split()))
print(l)
m= list(map(get_title_from_url, l))
print(m)
n= ' '.join(m)
print(n)

{Mozilla Firefox console}
errors:
Traceback (most recent call last):
File "<input>", line 10, in <module>
File "<input>", line 19, in get_title_from_url
AttributeError: 'NoneType' object has no attribute 'text'
NoneType is coming from that twitter URL for some reason. Even https://twitter.com leads to same error. Whereas, https://google.com works just fine. If I replace it with Google then here is what I get:
A new virus spreading in China has been detected in Seattle, says the CDC — making it the 1st U.S. case. ▪️ Health officials confirmed the virus is contagious ▪️ It has infected ~300 people, and killed 6, in China ▪️ Cases also in Japan, Thailand, Taiwan, South Korea Google
Can someone please help me figure out what's going on, and how can I fix that error? Thank you in advance. I looked around SO as best as I can, and couldn't relate anything to my issue.
EDIT: I have tried lxml method too, no luck on that either.
twit_link = "https://twitter.com"
link = requests.get(twit_link)
soup = BeautifulSoup(link.text,'lxml')
print(soup.find("title").get_text(strip=True))
errors like before.