Python: Beautifulsoup returning None or [ ]

user_10692834 0
Viewed 656

hello im practicing my requests and web scraping skills, so im attempting to scrape the trending page on youtube, and pull the title of the videos that are trending, which is this link youtube

this is the code im running

import requests
from bs4 import BeautifulSoup

url = 'https://www.youtube.com/feed/trending'
html = requests.get(url)
soup = BeautifulSoup(html.content, "html.parser")
a = soup.find_all("a", {"id": "video-title"})
print(a)

and its returning [], i dont understand why its returning [] when its the in the source code,

python-requests web web-scraping beautifulsoup python
3 Answers

print the contents of the variable html.content - does it contain that ID?

My bet is no, youtube.com is a heavily javascript dependant website, but the requests module doesn't have a js engine. What your browser sees usually isn't what a module like requests sees.

user_34343880

You may need a method like selenium which allows time for page to fully render. The following currently yields 70 titles.

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

url = 'https://www.youtube.com/feed/trending'

d = webdriver.Chrome()
d.get(url)
titles = [title.text for title in WebDriverWait(d,20).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "#video-title")))]
print(titles)
d.quit()
user_62412350

The web is devolving in that it's becoming increasingly inscrutable. "Modern" webpages, for the most part, are no longer generated by the server as the user will see them; rather, globs of script are being sent to the user and basically injecting whatever ¯\_(ツ)_/¯ into the DOM.

That's why you'll need to use Selenium bindings with a full-blown browser, as mentioned by QHarr above.

My apologies for not making this a comment, but apparently I need 50 points to do that.

user_108499890
Related