How to scrape a table with BeautifulSoup [Python]

Question

community,

I would like to deal with web scraping just for my own interest. The topic is very exciting and you can certainly gain a lot of interesting insights with the data available on the net.

In my first project I would like to generate some football data. For example, you can find information about previous injuries on Transfermarkt Leroy Sane.

Goal: To scrape the whole table.

Current status: I've managed to read out the Injuries column. Unfortunately I also get data from the "Missed matches" column.

Question: Is there a better way to get the exact values from each column? Or is there even a better way to scrape whole tables?

Code:

import requests
import csv
import re
from bs4 import BeautifulSoup

url = "https://www.transfermarkt.de/havard-nordtveit/verletzungen/spieler/192565"
heads = {'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36'}
response = requests.get(url, headers=heads)
html_icerigi = response.text
soup = BeautifulSoup(html_icerigi, "html.parser")
injuries = soup.find_all(class_="hauptlink")
injuries_list = []
for injury in injuries:
        injuries_list.append(injury.text)
print(injuries_list)

I am still at the beginning, you are the professionals. I am looking forward to learning.

Thanks, if you invest time to help me.

user_10987432 · Answer

The cells of the column you're interested in look like this: . However, the cells of the last column (missed games) are: . Your query will capture these unwanted cells because technically their class matches the one you're looking for, when what you really wanted was all td tags whose class match exactly hauptlink.

You can make your tag query more specific with a CSS selector that deliberately only looks for td tags whose class is exactly hauptlink:

def main():

    import requests
    from bs4 import BeautifulSoup

    url = "https://www.transfermarkt.de/havard-nordtveit/verletzungen/spieler/192565"

    headers = {
        "user-agent": "Mozilla/5.0"
    }

    response = requests.get(url, headers=headers)
    response.raise_for_status()

    soup = BeautifulSoup(response.content, "html.parser")

    for td in soup.select("td[class='hauptlink']"):
        print(td.text)

    return 0


if __name__ == "__main__":
    import sys
    sys.exit(main())

Also, if you want to just capture the text from the tags in a list, you could do something like this, which is a bit cuter than appending to a list in a loop:

second_column = list(map(lambda tag: tag.text, soup.select("td[class='hauptlink']")))

user_7752296 · Answer

You can get the table parsed with pandas (it uses beautifulsoup under the hood). .read_html() doesnt have to be a direct URL.

import requests
import pandas as pd

url = "https://www.transfermarkt.de/havard-nordtveit/verletzungen/spieler/192565"
heads = {'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36'}
response = requests.get(url, headers=heads)

# Instead of feeding into beautifulsoup to parse the table tags yourself,
# feed the html into pandas to do that for you 
df = pd.read_html(response.text)[0]

Output:

print(df.to_string())
  Saison              Verletzung         von         bis      Tage Verpasste Spiele
0  19/20           Kreuzbandriss  05.08.2019  08.03.2020  216 Tage               44
1  17/18              Bänderriss  29.01.2018  11.02.2018   13 Tage                3
2  17/18               Erkältung  28.11.2017  01.12.2017    3 Tage                1
3  16/17                Nasen-OP  23.05.2017  02.07.2017   40 Tage                -
4  16/17        Muskelverletzung  18.12.2016  13.01.2017   26 Tage                5
5  16/17  Oberschenkelverletzung  04.08.2016  04.09.2016   31 Tage                5
6  15/16          Handverletzung  07.09.2015  10.09.2015    3 Tage                -
7  14/15        kleinere Blessur  20.04.2015  23.04.2015    3 Tage                -
8  14/15        Muskelverletzung  14.07.2014  28.08.2014   45 Tage                4

user_12501078 · Answer

you can use this code to get all tables in your html file:

tables = soup.select('table', attrs={"class":"hauptlink"})

and get every table with its own index like:

my_table = tables[index]

and get tr tags like:

rows = my_table.findChildren(['tr'])

and print table's cells like:

for row in rows:
     cells = row.findChildren('td')
     print (  cells[1].text ... )

How to scrape a table with BeautifulSoup [Python]

3 Answers