beautifulsoup getting an AttributeError: 'NoneType' object has no attribute 'text' from a div's child's text

Question

The HTML looks a little like this: (there's a vast many more stats etc, but this is the first one)



53%
Ball Possession
47%

I'd like to draw out the 53% (without the % symbol if possible) as a home team value. Then I'd like to draw out the 47% (also without the % symbol if possible) as the away team value.

I can add the Ball Possession words later - or, if there's an easy way to append Ball Possession from this list, then I will do that too.

So, I did try this:

home_stats = soup.select_one('#statistics-content .statText--homeValue:nth-child(1)').text

And I searched the resultant error:

home_stats = soup.select_one('#statistics-content .statText--homeValue:nth-child(1)').text
AttributeError: 'NoneType' object has no attribute 'text'

But I can not seem to find something that fits this particular situation. My previous attempts have worked using this similar method, but they were either in a span or an a tag and had attributes. But this is just text wrapped in a div.

I can not change the HTML (sadly) - so I need to parse it directly from that HTML if possible.

The final outcome should look like:

[home_team] had 53% ball possession and [away_team] had 47% ball possession

Both home and away team works to give the correct name. I'd like "ball possession" to be all lower case and I'd like to add the % myself as I want to use those numbers separately later.

The code is all there, just not scraping the correct data. There's 14 other stats I'd like to retrieve, so hopefully, if I get this one right, then nth-child(x) should work.

Thanks kindly - this is very much my beginning journey of python and webscraping, so all help is appreciated.

To make it easier, I am scraping now from this site: https://www.scoreboard.com/en/match/SO3Fg7NR/#match-statistics;0

user_6106791 · Answer

Maybe this will get you started.

from bs4 import BeautifulSoup

sample_html = """

53%
Ball Possession
47%
"""


soup = BeautifulSoup(sample_html, "html.parser")

stats = [
    soup.select_one(f'#statistics-content .statText--{value}').getText(strip=True)
    for value in ["homeValue", "awayValue"]
]
outcome = "[home_team] had {} ball possession and [away_team] had {} ball possession".format(*stats)
print(outcome)

Output:

[home_team] had 53% ball possession and [away_team] had 47% ball possession

EDIT:

Thanks for the URL! It turns out there's nothing wrong with your code. The content of the page is behind JS, so BeautifulSoup won't see it. However, there are other ways to get what you're after.

One way of doing this, would be with selenium and chrome driver. In other words, this is an automation framework which allows us to execute test across various browsers or... scrape websites.

Here's what I've come up with:

import time

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from tabulate import tabulate


options = Options()
options.headless = False
driver = webdriver.Chrome(options=options)

# Use the web browser to access the url
driver.get("https://www.scoreboard.com/en/match/SO3Fg7NR/#match-statistics;0")
# wait for all the elements to load
time.sleep(2)

# Use BS4 to parse the page source obtained by the driver
soup = BeautifulSoup(driver.page_source, "html.parser")

# Close the web browser
driver.close()


# This is used to build a list of text values from all tags
def grab_tags(css_selector: str) -> list:
    return [t.getText(strip=True) for t in soup.select(css_selector)]


headers = grab_tags(".detail-experimental .statText--titleValue")
home_team = grab_tags(".detail-experimental .statText--homeValue")
away_team = grab_tags(".detail-experimental .statText--awayValue")

# glue all text items together per stat tab
# A list of tuples -> [(Ball Possesion, 53%, 47%), (Goal Attempts, 10, 5)...]
match_data = list(zip(headers, home_team, away_team))

split_value = 15
# split match data list stat table -> there are 15 items per table -> 3 tables
statistics = [
    match_data[i:i+split_value] for i in range(0, len(match_data), split_value)
]

# three stat tables
match, first_half, second_half = statistics

# print any of them
print(tabulate(match, headers=["Stat:", "Home Team", "Away Team"]))

Output:

Stat:              Home Team    Away Team
-----------------  -----------  -----------
Ball Possession    53%          47%
Goal Attempts      10           5
Shots on Goal      2            1
Shots off Goal     4            2
Blocked Shots      4            2
Free Kicks         11           10
Corner Kicks       8            2
Offsides           2            1
Goalkeeper Saves   0            2
Fouls              8            10
Yellow Cards       1            0
Total Passes       522          480
Tackles            15           12
Attacks            142          105
Dangerous Attacks  44           29

Note: I do realize this might blow your mind if you're at the beginning of your web-scraping journey, but I've put some comments to explain what's going on. Hope this helps!

user_10035985 · Answer

The data is loaded dynamically so Beautiful doesn't see the information inside the page. You can use this script to retrieve the data with BeautifulSoup:

import re
import requests
from bs4 import BeautifulSoup


url = 'https://www.scoreboard.com/en/match/SO3Fg7NR/#match-statistics;0'

headers = {'X-Fsign':'SW9D1eZo'}
match_id = re.search(r'match/([^/]+)/', url).group(1)
data_url = 'https://d.scoreboard.com/en/x/feed/d_st_{match_id}_en_1'.format(match_id=match_id)

soup = BeautifulSoup(requests.get(data_url, headers=headers).content, 'html.parser')

# locate row, that contains "Ball Possession":
row = soup.select_one('.statRow:has(*:contains("Ball Possession"))')

home_value = row.select_one('.statText--homeValue').text.replace('%', '')
away_value = row.select_one('.statText--awayValue').text.replace('%', '')

print(home_value, away_value)

Prints:

53 47

beautifulsoup getting an AttributeError: 'NoneType' object has no attribute 'text' from a div's child's text

2 Answers