Passing table data using Beautifulsoup

user_8836582 0
Viewed 720

I am trying to parse this webpage.

As shown below, each page has the ability stats. I am eventually trying to parse all abilities into an object. e.g. {'corners': 15, 'crossing': 15...}

I first started to parse a single stat, corners by doing:

from bs4 import BeautifulSoup as bs
import requests
url = 'https://fmdataba.com/19/p/1165/lionel-messi/'
page = requests.get(url)
soup = bs(page.content, 'html.parser')
print(soup.prettify())
soup.find({"id": "fm_cro"})

but this returns an empty list.

Could anyone please help?

enter image description here

web-scraping beautifulsoup html python
4 Answers

With bs4 4.7.1 you can use nth-child(odd) and nth-child(even) to get the different tds within each row to create your dict; and use :has and :contains to get the right table for each keyword and build your outer dict to house each inner.

import requests
from bs4 import BeautifulSoup as bs

r = requests.get('https://fmdataba.com/19/p/1165/lionel-messi/', headers = {'User-Agent':'Mozilla/5.0'})
soup = bs(r.content, 'lxml')
abilities = ['TECHNICAL', 'MENTAL' , 'PHYSICAL']

def get_abilities(soup, keyword):
    table = soup.select_one('div:has(h3:contains("' + ability + '")) + div > table')
    d = {item.select_one('td:nth-child(odd)').text: int(item.select_one('td:nth-child(even)').text) for item in table.select('tr')}
    return d

results = {}

for ability in abilities:
    results[ability] = get_abilities(soup, ability)

print(results)

Output:

enter image description here

CSS explanation:

The css selector line as follows:

soup.select_one('div:has(h3:contains("' + ability + '")) + div > table')

select_one is like select in that it applies the css selector within to the soup object but only returns the first match.

:has and :contains are pseudo classes like :nth-child(). Looking at the html in question for the first ability table here is an explanation of the parts:

Click on image to enlarge.

enter image description here

Additional reading:

  1. Pseudo class selectors
  2. Adjacent sibling combinator
  3. Child combinator
  4. Css selectors general
  5. select_one
user_62412350

You can also use pandas:

import pandas as pd
import requests

url = 'https://fmdataba.com/19/p/1165/lionel-messi/'
page = requests.get(url, headers={'User-Agent':'Mozilla/5.0'})

tables = pd.read_html(page.text)
all_data = {}
for idx, name in [(2, 'TECHNICAL'), (3, 'MENTAL'), (4, 'PHYSICAL')]:
    tbl = tables[idx]
    data = {r[0]: r[1] for _, r in tbl.iterrows()}
    all_data[name] = data

tables[2] is the TECHNICAL table, tables[3] is the MENTAL table and tables[4] is the PHYSICAL table.

user_20602050

Here is another way to achieve the same results without hardcoding indices to the selectors:

import requests
from bs4 import BeautifulSoup

r = requests.get('https://fmdataba.com/19/p/1165/lionel-messi/', headers={'User-Agent':'Mozilla/5.0'})
soup = BeautifulSoup(r.text, 'lxml')
datadict = {}
for container in ['TECHNICAL','MENTAL','PHYSICAL']:
    elem = soup.select_one(f".panel-heading:contains('{container}') + .panel-body")
    datadict[container] = {item.text:item.find_next_sibling().text for item in elem.select("td.active")}
print(datadict)

Output:

{'TECHNICAL': {'Corners': '15', 'Crossing': '15', 'Dribbling': '20', 'Finishing': '20', 'First Touch': '19', 'Free Kick': '19', 'Heading': '10', 'Long Shots': '17', 'Long Throws': '4', 'Marking': '4', 'Passing': '20', 'Penalty Taking': '17', 'Tackling': '7', 'Technique': '20'}, 'MENTAL': {'Aggression': '7', 'Anticipation': '19', 'Bravery': '10', 'Composure': '18', 'Concentration': '13', 'Decisions': '20', 'Determination': '20', 'Flair': '20', 'Leadership': '14', 'Off The Ball': '16', 'Positioning': '5', 'Teamwork': '14', 'Vision': '20', 'Work Rate': '7'}, 'PHYSICAL': {'Acceleration': '18', 'Agility': '20', 'Balance': '20', 'Jumping Reach': '6', 'Natural Fitness': '14', 'Pace': '15', 'Stamina': '13', 'Strength': '9'}}
user_105685310

Based on the documentation, the find() and find_all() methods do not take a dictionary, they have an optional id keyword argument you can use.

soup.find(id='fm_cro')

This will get you the element with that id (in this case the <td> with "Crossing" text). If you want to get the related number you'll need to use BeautifulSoup's sibling methods to get the next adjacent <td> and ask it for it's text (the number).

Soviut_469140
Related