Webscraping A Table Using Python and BeautifulSoup

user_14035209 0
Viewed 47

I'm learning on how to webscrape using Python since I'm a novice. Right now, I attempted to webscrape Euros 2020 stats from this website https://theanalyst.com/na/2021/06/euro-2020-player-stats. After running my initial code (see below) to gather the html from the webpage, I cannot locate the table tag and its data-table class. I can see the table and its data-table when I inspected the website, but it is not shown when I print out the page_soup.

from urllib.request import urlopen as uReq  # Web client
from bs4 import BeautifulSoup as soup  # HTML data structure

url_page = 'https://theanalyst.com/na/2021/06/euro-2020-player-stats'

# Open connection & download the html from the url
uClient = uReq(url_page)

# Parses html into a soup data structure
page_soup = soup(uClient.read(), "html.parser")
uClient.close()

print(page_soup)
web-scraping beautifulsoup python
1 Answers

The table is loaded dynamically in JSON format via sending a GET request to:

https://dataviz.theanalyst.com/euro-2020-hub/player_stats_3_2020.json

Since we're dealing with JSON data, it's easier to use the requests library to get the data.

Here is an example using the pandas library to print the table into a DataFrame (you don't have to use the pandas library).

import pandas as pd
import requests

url = "https://dataviz.theanalyst.com/euro-2020-hub/player_stats_3_2020.json"
response = requests.get(url).json()

print(pd.json_normalize(response["data"]).to_string())

Output (truncated):

    player_id  team_id        team_name           player_first_name           player_last_name                 player  age    position     detailed_position  mins_played  np_shots  np_sot  np_goals  np_xG  op_chances_created  op_assists  op_xA  op_passes  op_pass_completion_rate  tackles_won  interceptions  recoveries  avg_carry_distance  avg_carry_progress  carry_w_shot  carry_w_goal  carry_w_chance_created  carry_w_assist  take_ons  take_ons_success_rate  goal_ending  total_xG  shot_ending                                                                                                                                   team_badge
0       103955      114          England                      Raheem                   Sterling        Raheem Sterling   26     Forward        Second Striker          641        14       8         3   3.82                   2           1   1.18        193                     0.85            5              4          23               12.98                6.73             3             0                       3               1        38                  52.63            6      7.08           24  https://omo.akamai.opta.net/image.php?secure=true&h=omo.akamai.opta.net&sport=football&entity=team&description=badges&dimensions=150&id=114
1        56979      114          England                      Jordan                  Henderson       Jordan Henderson   31  Midfielder    Central Midfielder          150         1       1         1   0.32                   0           0   0.06        111                     0.88            0              1          11                7.83                0.49             0             0                       0               0         3                  66.67            0      0.00            0  https://omo.akamai.opta.net/image.php?secure=true&h=omo.akamai.opta.net&sport=football&entity=team&description=badges&dimensions=150&id=114
2        78830      114          England                       Harry                       Kane             Harry Kane   27     Forward               Striker          649        15       7         4   3.57                   5           0   0.39        159                     0.70            0              3           8               10.52                3.06             2             0                       2               0        15                  53.33            7      6.38           21  https://omo.akamai.opta.net/image.php?secure=true&h=omo.akamai.opta.net&sport=football&entity=team&description=badges&dimensions=150&id=114
3        58621      114          England                        Kyle                     Walker            Kyle Walker   31    Defender             Full Back          599         0       0         0   0.00                   2           0   0.18        352                     0.87            0              8          37               11.66                5.09             0             0                       0               0         1                 100.00            3      2.54           10  https://omo.akamai.opta.net/image.php?secure=true&h=omo.akamai.opta.net&sport=football&entity=team&description=badges&dimensions=150&id=114

The variable response is now a dictionary (dict) which you can access the keys/values. To view and prettify the data:

from pprint import pprint

print(type(response))
pprint(response)

Output (truncated):

<class 'dict'>
{'data': [{'age': 26,
           'avg_carry_distance': 12.98,
           'avg_carry_progress': 6.73,
           'carry_w_assist': 1,
           'carry_w_chance_created': 3,
           'carry_w_goal': 0,
           'carry_w_shot': 3,
           'detailed_position': 'Second Striker',
user_123497340
Related