Scraping Iframe Using BeautifulSoup

Question

I'm trying to scrape a live score from a dynamic website (https://www.sportingindex.com/spread-betting/cricket/international-odi-matches/group_c.37afeb89-7976-47fd-9f17-29c5777e43cd/2nd-odi-bangladesh-v-sri-lanka).

I know I need to fetch the URL in the iframe src and I thought the code below would do the job but I can't get it to work. The version below gives an error.

"TypeError: 'NoneType' object is not iterable" error.

What am I missing?

import requests
from bs4 import BeautifulSoup

s = requests.Session()
r = s.get("https://www.sportingindex.com/spread-betting/cricket/international-odi-matches/group_c.37afeb89-7976-47fd-9f17-29c5777e43cd/2nd-odi-bangladesh-v-sri-lanka")

soup = BeautifulSoup(r.content, "html.parser")
iframe_src = soup.find("iframe").attrs["src"]

r = s.get(f"https:{iframe_src}")
soup = BeautifulSoup(r.content, "html.parser")
for container in soup.find("sb-ms-score"):
    score = container.find("span", {"data-gp-sb-callback": "updateMainScore"}).text.strip()
    print(score)

Edit:

So it seems the iframe content is in javascript so I've tried to build in Selenium. The updated code is here:

import requests
from bs4 import BeautifulSoup
from selenium import webdriver

firefox_driver_path = 'geckodriver.exe'
driver = webdriver.Firefox(executable_path=firefox_driver_path)
s = requests.Session()
r = s.get("https://www.sportingindex.com/spread-betting/tennis/utr-pro-match-series-womens/group_b.87d1e4ea-be95-4bc4-99d4-47f105484ecd/r-kolzer-v-b-passola")

soup = BeautifulSoup(r.content, "html.parser")

score_section = soup.find('section', class_='scoreboard gp')

iframe_src = score_section.find("iframe").attrs["src"]

r2 = driver.get(iframe_src)

soup2 = BeautifulSoup(r2.content, "html.parser")

pp = soup2.find('div', class_='sb-ms-inplay')

print(pp)

This throws up the error below at this line: soup2 = BeautifulSoup(r2.content, "html.parser"). This seems strange to me because when I print iframe_src it returns the iframe url correctly.

Traceback (most recent call last):
  File "C:/SportingIndex v9.py", line 18, in 
    soup2 = BeautifulSoup(r2.content, "html.parser")
AttributeError: 'NoneType' object has no attribute 'content'

Note the URL is different as yesterday's game is no longer live.

Can anyone see the problem?

user_2610022 · Answer

There appear to be a couple of problems with your approach, but the main one is your use of soup.find().

This code should illustrate:

>>> r = s.get("https://www.sportingindex.com/spread-betting/cricket/international-odi-matches/group_c.37afeb89-7976-47fd-9f17-29c5777e43cd/2nd-odi-bangladesh-v-sri-lanka")
>>> soup = BeautifulSoup(r.content, "html.parser")
>>> soup.find("iframe")

>>> soup.find_all("iframe")
[, ]

i.e. find() returns one result, whereas find_all() returns all.

user_4787888 · Answer

The snippet below gets you a step further, but the content you are looking for "sb-ms-score" is not in the urls you are looking up

import requests 
from bs4 import BeautifulSoup 
 
s = requests.Session() 
r = s.get("https://www.sportingindex.com/spread-betting/cricket/international-odi-matches/group_c.37afeb89-7976-47fd-9f17-29c5777e43cd/2nd-odi-bangladesh-v-sri-lanka") 
 
soup = BeautifulSoup(r.content, "html.parser") 
iframe_src = soup.find_all("iframe") 
 
for ifr in iframe_src: 
    _src_str = "{:}".format(ifr["src"]) 
    if "https" not in _src_str: 
        _url = "https:{:}".format(iframe_src[0]["src"]) 
    else: 
        _url = _src_str 
    r = s.get(_url) 
    print(_url) 
    print(r) 
    _soup = BeautifulSoup(r.content, "html.parser") 
    for container in _soup.find_all("sb-ms-score"): 
        print(container)

Scraping Iframe Using BeautifulSoup

2 Answers