Python web-scraping misses an element from the list of searched objects

user_13562029 0
Viewed 63

I'm trying to scrape some data using beautifulsoup and requests libraries in Python 3.7. For each of the items (tag article) on this webpage, there is a youtube link. After finding all the instances of article, I can successfully extract the headlines. This code also successfully finds instances of youtube-player class inside each article, except at index 7, where the output is None.

from bs4 import BeautifulSoup
import requests
url = 'https://coreyms.com/page/12'
soup = BeautifulSoup(requests.get(url).text, "html.parser")
articles = soup.find_all('article')

for article in articles:
    headline = article.h2.a.text
    print(headline)
    link = article.find('iframe', {'class': 'youtube-player'})
    print(link)

However, from the source (output of beautifulsoup), if I directly search for youtube-player, I get all the instances correctly.

links = soup.find_all('iframe', {'class': 'youtube-player'})
for link in links:
    print(link)

How can I improve my code to get all the youtube-player instances within article loop?

python-requests web-scraping beautifulsoup python-3.x
1 Answers

You can use zip() built-in function to tie titles and youtube links together.

For example:

import requests
from bs4 import BeautifulSoup

url = 'https://coreyms.com/page/12'
soup = BeautifulSoup(requests.get(url).text, "html.parser")

for title, player in zip(soup.select('.entry-title'),
                         soup.select('iframe.youtube-player')):
    print('{:<75}{}'.format(title.text, player['src']))

Prints:

Git: Difference between “add -A”, “add -u”, “add .”, and “add *”           https://www.youtube.com/embed/tcd4txbTtAY?version=3&rel=1&fs=1&autohide=2&showsearch=0&showinfo=1&iv_load_policy=1&wmode=transparent
Programming Terms: Combinations and Permutations                           https://www.youtube.com/embed/QI9EczPQzPQ?version=3&rel=1&fs=1&autohide=2&showsearch=0&showinfo=1&iv_load_policy=1&wmode=transparent
Chrome Quick Tip: Quickly Bookmark Open Tabs for Later Viewing             https://www.youtube.com/embed/tsiSg_beudo?version=3&rel=1&fs=1&autohide=2&showsearch=0&showinfo=1&iv_load_policy=1&wmode=transparent
Python: Comprehensions – How they work and why you should be using them    https://www.youtube.com/embed/3dt4OGnU5sM?version=3&rel=1&fs=1&autohide=2&showsearch=0&showinfo=1&iv_load_policy=1&wmode=transparent
Python: Generators – How to use them and the benefits you receive          https://www.youtube.com/embed/bD05uGo_sVI?version=3&rel=1&fs=1&autohide=2&showsearch=0&showinfo=1&iv_load_policy=1&wmode=transparent
Quickest and Easiest Way to Run a Local Web-Server                         https://www.youtube.com/embed/lE6Y6M9xPLw?version=3&rel=1&fs=1&autohide=2&showsearch=0&showinfo=1&iv_load_policy=1&wmode=transparent
Git for Beginners: Command-Line Fundamentals                               https://www.youtube.com/embed/HVsySz-h9r4?version=3&rel=1&fs=1&autohide=2&showsearch=0&showinfo=1&iv_load_policy=1&wmode=transparent
Time-Saving Keyboard Shortcuts for the Mac Terminal                        https://www.youtube.com/embed/TXzrk3b9sKM?version=3&rel=1&fs=1&autohide=2&showsearch=0&showinfo=1&iv_load_policy=1&wmode=transparent
Overview of Online Learning Resources in 2015                              https://www.youtube.com/embed/QGy6M8HZSC4?version=3&rel=1&fs=1&autohide=2&showsearch=0&showinfo=1&iv_load_policy=1&wmode=transparent
Python: Else Clauses on Loops                                              https://www.youtube.com/embed/Dh-0lAyc3Bc?version=3&rel=1&fs=1&autohide=2&showsearch=0&showinfo=1&iv_load_policy=1&wmode=transparent

EDIT: It seems that when you use html.parser, BeautifulSoup doesn't recognize the youtube link on one place, use lxml or html5lib instead:

import requests
from bs4 import BeautifulSoup

url = 'https://coreyms.com/page/12'
soup = BeautifulSoup(requests.get(url).text, "lxml")

for article in soup.select('article'):
    title = article.select_one('.entry-title')
    player = article.select_one('iframe.youtube-player') or {'src':''}
    print('{:<75}{}'.format(title.text, player['src']))
user_100359850
Related