twitter url that exists returns None with Beautiful Soup

Viewed 254

Hello I am trying to replace url links from string with that url's title tag. Here is the function:

import requests
import urllib.request
import re
from bs4 import BeautifulSoup as bs
import string

def get_title_from_url(url):

    #check if it's a word of https string
    if(url.startswith('http')):
        #check if url is valid
        try:
            page = requests.get(url)
        except:
            return ''

        #if the url is valid, sometimes phisy/ssl-missing site throw error
        try:
            html = bs(page.content, 'html.parser')
        except:
            return ''

        #finally get the title if everyting is good
        page_title = html.find('title').text.strip()

        return page_title
    else:
        return url

then here is a sample string that gets me the error even though the URL for tweet exists and leads to the AJ news.

s = 'A new virus spreading in China has been detected in Seattle, ' \
    'says the CDC — making it the 1st U.S. case.\r\n\r\n▪️ Health officials ' \
    'confirmed the virus is contagious\r\n▪️ It has infected ~300 people, and killed ' \
    '6, in China\r\n▪️ Cases also in Japan, Thailand, Taiwan, ' \
    'South Korea https://twitter.com' #original link to tweet removed due to SO guideline

l = list(map(lambda x: x, s.split()))
print(l)
m= list(map(get_title_from_url, l))
print(m)
n= ' '.join(m)
print(n)

enter image description here
{Mozilla Firefox console}
errors:

Traceback (most recent call last):
  File "<input>", line 10, in <module>
  File "<input>", line 19, in get_title_from_url
AttributeError: 'NoneType' object has no attribute 'text'

NoneType is coming from that twitter URL for some reason. Even https://twitter.com leads to same error. Whereas, https://google.com works just fine. If I replace it with Google then here is what I get:

A new virus spreading in China has been detected in Seattle, says the CDC — making it the 1st U.S. case. ▪️ Health officials confirmed the virus is contagious ▪️ It has infected ~300 people, and killed 6, in China ▪️ Cases also in Japan, Thailand, Taiwan, South Korea Google

Can someone please help me figure out what's going on, and how can I fix that error? Thank you in advance. I looked around SO as best as I can, and couldn't relate anything to my issue.

EDIT: I have tried lxml method too, no luck on that either.

twit_link = "https://twitter.com"

link = requests.get(twit_link)
soup = BeautifulSoup(link.text,'lxml')
print(soup.find("title").get_text(strip=True))

errors like before.

1 Answers

So I have figured it out. Twitter's tweet url are considered private information, therefore, any type of web-scapper can't open it.

Related