Python web scraping leaving gaps in MLB pull

user_19684396 0
Viewed 77

I am trying to scrape MLB lineups from this link: https://www.baseballpress.com/lineups/2022-08-04 using Python. However, when I run the code below, it pulls the data into a dataframe, with the away teams in the "first_lineup" column and the home teams in the "second_lineup" column. The issue is that when it pulls the data into a the dataframe, it does not pull everyone. Attached are screenshots of the data frame and of the website. If anyone could help explain the issue, that would be great! Thanks.

Dataframe Code Output

Website example

import requests
import pandas as pd
from bs4 import BeautifulSoup


url = "https://www.baseballpress.com/lineups/2022-08-04"

def get_names(item):
    try:
        player_name = item.get('data-razz').split("/")[-2].replace("+"," ")
    except IndexError: player_name = ""
    return player_name

r = requests.get(url)
soup = BeautifulSoup(r.text,'lxml')
first_lineup = [get_names(item) for item in soup.select(".col--min:nth-of-type(1) > a.player-link, [class$='col--min']:nth-of-type(1) .player > a.player-link")]
second_lineup = [get_names(item) for item in soup.select(".col--min:nth-of-type(2) > a.player-link, [class$='col--min']:nth-of-type(2) .player > a.player-link")]

df = pd.DataFrame({"first_lineup":first_lineup,"second_lineup":second_lineup})
df.to_csv("baseballpress.csv", encoding='utf-8', index=False)
print(df)
pandas web-scraping python
2 Answers

To get all players to a dataframe you can use next example:

import requests
import pandas as pd
from bs4 import BeautifulSoup


url = "https://www.baseballpress.com/lineups/2022-08-04"
soup = BeautifulSoup(requests.get(url).content, "html.parser")


def get_name(tag):
    if tag.select_one(".mobile-name"):
        tag.select_one(".mobile-name").extract()
    return tag


data = []
for card in soup.select(".lineup-card"):
    header = [
        c.get_text(strip=True, separator=" ")
        for c in card.select(".lineup-card-header .c")
    ]
    h_p1, h_p2 = [
        get_name(p).text for p in card.select(".lineup-card-header .player")
    ]
    data.append([*header, h_p1, h_p2])

    for p1, p2 in zip(
        card.select(".col--min:nth-of-type(1) .player"),
        card.select(".col--min:nth-of-type(2) .player"),
    ):
        p1 = get_name(p1).text.split(maxsplit=1)[-1]
        p2 = get_name(p2).text.split(maxsplit=1)[-1]

        data.append([*header, p1, p2])

df = pd.DataFrame(
    data, columns=["Team1", "Date", "Team2", "Player1", "Player2"]
)
df.to_csv("data.csv", index=False)
print(df.head(10).to_markdown(index=False))

Prints:

Team1 Date Team2 Player1 Player2
Brewers August, 4 12:35pm Pirates Brandon Woodruff (R) Zach Thompson (R)
Brewers August, 4 12:35pm Pirates Christian Yelich (L) DH Tucupita Marcano (L) 2B
Brewers August, 4 12:35pm Pirates Willy Adames (R) SS Bryan Reynolds (S) DH
Brewers August, 4 12:35pm Pirates Rowdy Tellez (L) 1B Ben Gamel (L) LF
Brewers August, 4 12:35pm Pirates Andrew McCutchen (R) LF Ke'Bryan Hayes (R) 3B
Brewers August, 4 12:35pm Pirates Kolten Wong (L) 2B Oneil Cruz (L) SS
Brewers August, 4 12:35pm Pirates Hunter Renfroe (R) RF Bligh Madris (L) 1B
Brewers August, 4 12:35pm Pirates Victor Caratini (S) C Greg Allen (S) CF
Brewers August, 4 12:35pm Pirates Mike Brosseau (R) 3B Cal Mitchell (L) RF
Brewers August, 4 12:35pm Pirates Jonathan Davis (R) CF Jason Delay (R) C

and saves data.csv (screenshot from LibreOffice):

enter image description here

user_100359850

You are using the data-razz property for each player in your listing, but some of those are empty for some players like Seiya Suzuki, if you take a look at the HTML, you'll see his and Nico Hoerner's are missing this field.\

 <a class="player-link" data-bref="wisdopa01" data-mlb="621550" data-razz="https://razzball.com/player/13602/Patrick+Wisdom/" href="http://mlb.mlb.com/team/player.jsp?player_id=621550" target="_blank">Patrick Wisdom</a>
 <a class="player-link" data-bref="suzukse01" data-mlb="673548" data-razz="" href="http://mlb.mlb.com/team/player.jsp?player_id=673548" target="_blank">Seiya Suzuki</a>
 <a class="player-link" data-bref="hoernni01" data-mlb="663538" data-razz="" href="http://mlb.mlb.com/team/player.jsp?player_id=663538" target="_blank">Nico Hoerner</a>

Similarly, a number of players, probably due to name length, have additional selectors for desktop-name (which contains text for a full name, e.g. Brandon Woodruff) and mobile-name (containing a shortened name, e.g. B. Woodruff), like so:

<a class="player-link" data-bref="woodrbr01" data-mlb="605540" data-razz="https://razzball.com/player/605540/Brandon+Woodruff/" href="http://mlb.mlb.com/team/player.jsp?player_id=605540" target="_blank">
    <span class="desktop-name">Brandon Woodruff</span>
    <span class="mobile-name">B. Woodruff</span>
</a>

It doesn't look like your code as-is particularly cares about the actual razzball URL, so you could call get_text() on each field to get the player name as displayed via the <p> tags, with some additional checking to see if the player has a desktop-name field.

from bs4 import Tag

# ...stuff...
def get_player_name(t: Tag) -> str:
    if span := t.find("span", {"class": "desktop-name"}):
        return span.get_text()
    
    return t.get_text()

And then call it...

first_lineup = [get_player_name(t) for t in soup.select(".col--min:nth-of-type(1) > a.player-link, [class$='col--min']:nth-of-type(1) .player > a.player-link")]
second_lineup = [get_player_name(t) for t in soup.select(".col--min:nth-of-type(2) > a.player-link, [class$='col--min']:nth-of-type(2) .player > a.player-link")]

And now you should have the longer names for every player.

user_2976960
Related