How to add values from different dataframes together?

Viewed 40

I didn't think it would be this complicated, but I want to do basic math from a value from a cell in one dataframe with another value from a cell in another dataframe.

These are the dataframes I am using

url = 'https://www.basketball-reference.com/leagues/NBA_2021.html'
page = requests.get(url)
soup = BeautifulSoup(page.content, 'html.parser')
table = soup.find('table', id='per_game-team')
teamdf = pd.read_html(str(table))[0]
teamdf

and

url = 'https://www.basketball-reference.com/leagues/NBA_2021.html'
page = requests.get(url)
soup = BeautifulSoup(page.content, 'html.parser')
table = soup.find('table', id='advanced-team')
advdf = pd.read_html(str(table))[0]
advdf.columns = advdf.columns.map('_'.join)
advdf.rename_axis('Rk').reset_index()
advdf.drop(columns=['Unnamed: 17_level_0_Unnamed: 17_level_1', 'Unnamed: 22_level_0_Unnamed: 22_level_1', 'Unnamed: 27_level_0_Unnamed: 27_level_1', 'Unnamed: 28_level_0_Arena', 'Unnamed: 29_level_0_Attend.', 'Unnamed: 30_level_0_Attend./G'], inplace=True)
advdf.rename(
    columns={
        'Unnamed: 0_level_0_Rk': 'Rk',
        'Unnamed: 1_level_0_Team': 'Team',
        'Unnamed: 2_level_0_Age': 'Age',
        'Unnamed: 3_level_0_W': 'W',
        'Unnamed: 4_level_0_L': 'L',
        'Unnamed: 5_level_0_PW': 'PW',
        'Unnamed: 6_level_0_PL': 'PL',
        'Unnamed: 7_level_0_MOV': 'MOV',
        'Unnamed: 8_level_0_SOS': 'SOS',
        'Unnamed: 9_level_0_SRS': 'SRS',
        'Unnamed: 10_level_0_ORtg': 'ORtg',
        'Unnamed: 11_level_0_DRtg': 'DRtg',
        'Unnamed: 12_level_0_NRtg': 'NRtg',
        'Unnamed: 13_level_0_Pace': 'Pace',
        'Unnamed: 14_level_0_FTr': 'FTr',
        'Unnamed: 15_level_0_3PAr': '3PAr',
        'Unnamed: 16_level_0_TS%': 'TS%',
        'Offense Four Factors_eFG%': 'O_eFG%',
        'Offense Four Factors_TOV%': 'O_TOV%',
        'Offense Four Factors_ORB%': 'O_ORB%',
        'Offense Four Factors_FT/FGA': 'O_FT/FGA',
        'Defense Four Factors_eFG%': 'D_eFG%',
        'Defense Four Factors_TOV%': 'D_TOV%',
        'Defense Four Factors_DRB%': 'D_DRB%',
        'Defense Four Factors_FT/FGA': 'D_FT/FGA'
    }, inplace=True)
advdf

So I filter by team, and select the values from there

team1df = teamdf[teamdf.Team == 'Philadelphia 76ers*']
team1adf = advdf[advdf.Team == 'Philadelphia 76ers*']

If I wanted to multiply any values, for this example the their field goal percentage and offensive rating, I would do

t1_fgp = team1df['FG%']
t1a_ortg = team1adf['ORtg']

t1_fgp*t1a_ortg

But instead of giving me the value I want, output looks like

Output:
4    NaN
13   NaN
dtype: float64

Please help this is frustrating.

2 Answers

The problem is that the 4 and 13 are the indices of 76ers in the corresponding tables. When you try to multiply series, pandas tries to match the values by the given index.

Instead of multiplying the series directly you could try:

t1_fgp = team1df['FG%']
t1a_ortg = team1adf['ORtg']

t1a_ortg.iloc[0]*t1_fgp.iloc[0]

Output:

53.8832

Edit:

Or a better way might be merging the dataframes:

# Teamdf: Rename the rk column to teamdfrk
teamdf.rename(columns={'Rk': 'teamdfrk'}, inplace=True)

# Advdf: Rename the rk column to advrk
advdf.rename(columns={'Rk': 'advrk'}, inplace=True)

# Merge the two dataframes based on the Team column
mergeddf = pd.merge(teamdf, advdf, on='Team')

Then use it accordingly:

phil = mergeddf[mergeddf.Team == 'Philadelphia 76ers*']
phil['FG%'] * phil['ORtg']

Output:

13    53.8832
dtype: float64

As stated, the index doesn't match. Another option is to reset the index first:

team1df = teamdf[teamdf.Team == 'Philadelphia 76ers*'].reset_index(drop=True)
team1adf = advdf[advdf.Team == 'Philadelphia 76ers*'].reset_index(drop=True)

Personally, I'd merge the two dataframes, then can just apply that calulation on the entire columns.

Also, I cleaned up your code a bit. It's over kill here to use requests and beautifulsoup, as pandas uses those both in the .read_html()

Code:

import pandas as pd

url = 'https://www.basketball-reference.com/leagues/NBA_2021.html'
teamdf = pd.read_html(url, attrs={'id':'per_game-team'})[0]

url = 'https://www.basketball-reference.com/leagues/NBA_2021.html'
advdf = pd.read_html(url, attrs={'id':'advanced-team'}, header=1)[0]

full_df = pd.merge(teamdf, advdf.drop('Rk', axis=1), on=['Team'])
full_df['calculated_column'] = full_df['FG%'] * full_df['ORtg'] 

Output:

print(full_df)
      Rk                     Team   G  ...   Attend.  Attend./G  calculated_column
0    1.0         Milwaukee Bucks*  72  ...   64780.0     1799.0            57.0764
1    2.0           Brooklyn Nets*  72  ...   30491.0      847.0            58.4402
2    3.0      Washington Wizards*  72  ...   19198.0      533.0            52.8200
3    4.0               Utah Jazz*  72  ...  151300.0     4203.0            55.0368
4    5.0  Portland Trail Blazers*  72  ...    5817.0      162.0            53.3634
5    6.0            Phoenix Suns*  72  ...  104027.0     2890.0            57.4280
6    7.0           Indiana Pacers  72  ...       NaN        NaN            53.2776
7    8.0          Denver Nuggets*  72  ...   54563.0     1516.0            56.7935
8    9.0     New Orleans Pelicans  72  ...   93120.0     2587.0            54.1395
9   10.0    Los Angeles Clippers*  72  ...   13901.0      386.0            56.6832
10  11.0           Atlanta Hawks*  72  ...   59288.0     1647.0            54.1476
11  12.0         Sacramento Kings  72  ...       NaN        NaN            54.6416
12  13.0    Golden State Warriors  72  ...   33457.0      929.0            51.9948
13  14.0      Philadelphia 76ers*  72  ...   68583.0     1905.0            53.8832
14  15.0       Memphis Grizzlies*  72  ...   61449.0     1707.0            52.3040
15  16.0          Boston Celtics*  72  ...   30067.0      835.0            53.1240
16  17.0        Dallas Mavericks*  72  ...   94849.0     2635.0            54.2380
17  18.0   Minnesota Timberwolves  72  ...   15774.0      438.0            49.0560
18  19.0          Toronto Raptors  72  ...   26024.0      723.0            50.1760
19  20.0        San Antonio Spurs  72  ...   61053.0     1696.0            51.2820
20  21.0            Chicago Bulls  72  ...   13655.0      379.0            52.8836
21  22.0      Los Angeles Lakers*  72  ...   23313.0      648.0            51.8728
22  23.0        Charlotte Hornets  72  ...   68255.0     1896.0            50.4595
23  24.0          Houston Rockets  72  ...  117009.0     3250.0            47.5524
24  25.0              Miami Heat*  72  ...       NaN        NaN            52.0416
25  26.0         New York Knicks*  72  ...   42131.0     1170.0            50.4336
26  27.0          Detroit Pistons  72  ...   14250.0      396.0            48.8160
27  28.0    Oklahoma City Thunder  72  ...       NaN        NaN            45.6435
28  29.0            Orlando Magic  72  ...  126463.0     3513.0            45.0879
29  30.0      Cleveland Cavaliers  72  ...   91476.0     2541.0            47.6100
30   NaN           League Average  72  ...   49476.0     1374.0            52.3318
Related