Find_next('table') BeautifulSoup

user_12113958 0
Viewed 62

I would to scrape tables with the headers contains a word

import requests
from bs4 import BeautifulSoup

url ='https://www.transfermarkt.com/juventus-turin/alletransfers/verein/506'
headers = {
        "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0"
    }
content = requests.get(url, headers=headers)
soup = BeautifulSoup(content.text, 'html.parser')

h2 = soup.find_all(lambda elm: elm.name == 'h2' and 'Departures' in elm.text)
for h in h2:
    table = h.find_next('table')
    print(table)

With this code I get 'Arrivals' tables. How can I get only the Departures tables to work on?

web-scraping beautifulsoup python
2 Answers

You can select all <table>s and then filter the correct one according the header:

import requests
from bs4 import BeautifulSoup

url = "https://www.transfermarkt.com/juventus-turin/alletransfers/verein/506"
headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0"
}
content = requests.get(url, headers=headers)
soup = BeautifulSoup(content.text, "html.parser")


tables = soup.select(".table-header + table")
for table in tables:
    prev_h2 = table.find_previous("h2")
    if not "Departures" in prev_h2.text:
        continue

    # print some data from the table:
    print(prev_h2.get_text(strip=True))
    print("-" * 80)
    for tr in table.select("tr:has(td)"):
        row = [td.get_text(strip=True) for td in tr.select("td")[:3]]
        if len(row) != 3:
            continue
        link = "https://www.transfermarkt.com" + tr.a["href"]
        print("{:<30} {:<20} {:<20} {}".format(*row, link))
    print()

Prints:

Departures 20/21
--------------------------------------------------------------------------------
Miralem Pjanić                                      FC Barcelona         https://www.transfermarkt.com/miralem-pjani%C4%87/profil/spieler/44162
Emre Can                                            Bor. Dortmund        https://www.transfermarkt.com/emre-can/profil/spieler/119296
Manolo Portanova                                    Genoa                https://www.transfermarkt.com/manolo-portanova/profil/spieler/394304
Simone Muratore                                     Atalanta BC          https://www.transfermarkt.com/simone-muratore/profil/spieler/297648
Cristian Romero                                     Atalanta BC          https://www.transfermarkt.com/cristian-romero/profil/spieler/355915
Daniele Rugani                                      Stade Rennais        https://www.transfermarkt.com/daniele-rugani/profil/spieler/162959
Matteo Stoppa                                       Sampdoria            https://www.transfermarkt.com/matteo-stoppa/profil/spieler/391899
Daniele Rugani                                      Cagliari Calcio      https://www.transfermarkt.com/daniele-rugani/profil/spieler/162959
Douglas Costa                                       FC Bayern            https://www.transfermarkt.com/douglas-costa/profil/spieler/75615
Stefano Gori                                        AC Pisa              https://www.transfermarkt.com/stefano-gori/profil/spieler/244491
Rolando Mandragora                                  Torino               https://www.transfermarkt.com/rolando-mandragora/profil/spieler/308279
Rolando Mandragora                                  Udinese Calcio       https://www.transfermarkt.com/rolando-mandragora/profil/spieler/308279
Gonzalo Higuaín                                     Inter Miami CF       https://www.transfermarkt.com/gonzalo-higuain/profil/spieler/39153
Blaise Matuidi                                      Inter Miami CF       https://www.transfermarkt.com/blaise-matuidi/profil/spieler/33923
Sami Khedira                                        Hertha BSC           https://www.transfermarkt.com/sami-khedira/profil/spieler/29401
Marko Pjaca                                         Genoa                https://www.transfermarkt.com/marko-pjaca/profil/spieler/205940
Luca Pellegrini                                     Genoa                https://www.transfermarkt.com/luca-pellegrini/profil/spieler/346567
Nicolò Rovella                                      Genoa                https://www.transfermarkt.com/nicolo-rovella/profil/spieler/525704
Hans Nicolussi Caviglia                             Parma                https://www.transfermarkt.com/hans-nicolussi-caviglia/profil/spieler/430280
Mattia De Sciglio                                   Olympique Lyon       https://www.transfermarkt.com/mattia-de-sciglio/profil/spieler/88682

Departures 19/20
--------------------------------------------------------------------------------
João Cancelo                                        Man City             https://www.transfermarkt.com/joao-cancelo/profil/spieler/182712
Leonardo Spinazzola                                 AS Roma              https://www.transfermarkt.com/leonardo-spinazzola/profil/spieler/118689
Moise Kean                                          Everton              https://www.transfermarkt.com/moise-kean/profil/spieler/364135
Emil Audero                                         Sampdoria            https://www.transfermarkt.com/emil-audero/profil/spieler/256339
Stefano Sturaro                                     Genoa                https://www.transfermarkt.com/stefano-sturaro/profil/spieler/167859
Riccardo Orsolini                                   Bologna              https://www.transfermarkt.com/riccardo-orsolini/profil/spieler/368482
Alberto Cerri                                       Cagliari Calcio      https://www.transfermarkt.com/alberto-cerri/profil/spieler/197751
Andrea Favilli                                      Genoa                https://www.transfermarkt.com/andrea-favilli/profil/spieler/340151
Rogério                                             Sassuolo             https://www.transfermarkt.com/rogerio/profil/spieler/401527
Leonardo Mancuso                                    FC Empoli            https://www.transfermarkt.com/leonardo-mancuso/profil/spieler/160344
Emre Can                                            Bor. Dortmund        https://www.transfermarkt.com/emre-can/profil/spieler/119296
Mattia Perin                                        Genoa                https://www.transfermarkt.com/mattia-perin/profil/spieler/110923
Luca Marrone                                        Hellas Verona        https://www.transfermarkt.com/luca-marrone/profil/spieler/124769
Luca Barlocco                                       LR Vicenza           https://www.transfermarkt.com/luca-barlocco/profil/spieler/197741
Stefano Padovan                                     Imolese              https://www.transfermarkt.com/stefano-padovan/profil/spieler/30263
Filippo Marricchi                                   Novara               https://www.transfermarkt.com/filippo-marricchi/profil/spieler/355027
Dejan Kulusevski                                    Parma                https://www.transfermarkt.com/dejan-kulusevski/profil/spieler/431755
Mario Mandzukic                                     Al-Duhail SC         https://www.transfermarkt.com/mario-mandzukic/profil/spieler/34572
Cristian Romero                                     Genoa                https://www.transfermarkt.com/cristian-romero/profil/spieler/355915
Marko Pjaca                                         RSC Anderlecht       https://www.transfermarkt.com/marko-pjaca/profil/spieler/205940
Luca Pellegrini                                     Cagliari Calcio      https://www.transfermarkt.com/luca-pellegrini/profil/spieler/346567
Andrea Barzagli                                     Retired              https://www.transfermarkt.com/andrea-barzagli/profil/spieler/7109

...and so on.

UPDATED: Added links to players.

user_100359850

You can use :has and :contains to target the right tables, by specifying the element with class table-header, has a child h2 which contains "Departures", then move to the adjacent sibling table element.

import requests
from bs4 import BeautifulSoup as bs
import pandas as pd

base = 'https://www.transfermarkt.com'
headers = {"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0"}
r = requests.get('https://www.transfermarkt.com/juventus-turin/alletransfers/verein/506', headers = headers )
soup = bs(r.content, 'lxml')
tables = soup.select('.table-header:has(h2:contains("Departures")) + table')
dfs = []

for table in tables:
    player_links = [base + i['href'] for i in table.select('.spielprofil_tooltip')]
    team_links = [base + i['href'] for i in table.select('.zentriert > .vereinprofil_tooltip')]
    df = pd.read_html(str(table))[0].iloc[:-1, :]
    df['Club'] = team_links
    df['Profile Link'] = player_links
    df.rename(columns={'Club':'Club Link', 'Club.1':'Club Name'}, inplace=True)
    df.dropna(how='all', axis=1, inplace=True)
    df = df[['Players', 'Profile Link', 'Club Name', 'Club Link', 'Transfer sum']]
    print(df)
    dfs.append(df)
user_62412350
Related