Python/Pandas: Use lookup DataFrame + function to replace specific/null values in DataFrame

user_2606407 0
Viewed 304

Say I have an incomplete dataset in a Pandas DataFrame such as:

incData  = pd.DataFrame({'comp': ['A']*3 + ['B']*5 + ['C']*4,
                         'x': [1,2,3] + [1,2,3,4,5] + [1,2,3,4],
                         'y': [3,None,7] + [1,4,7,None,None] + [4,None,2,1]})

And also a DataFrame with fitting parameters that I could use to fill holes:

fitTable = pd.DataFrame({'slope': [2,3,-1],
                         'intercept': [1,-2,5]},
                         index=['A','B','C'])

I would like to achieve the following using y=x*slope+intercept for the None entries only:

   comp  x     y
0     A  1   3.0
1     A  2   5.0
2     A  3   7.0
3     B  1   1.0
4     B  2   4.0
5     B  3   7.0
6     B  4  10.0
7     B  5  13.0
8     C  1   4.0
9     C  2   3.0
10    C  3   2.0
11    C  4   1.0

One way I envisioned is by using join and drop:

incData = incData.join(fitTable,on='comp')
incData.loc[incData['y'].isnull(),'y'] = incData[incData['y'].isnull()]['x']*\
                                         incData[incData['y'].isnull()]['slope']+\
                                         incData[incData['y'].isnull()]['intercept']
incData.drop(['slope','intercept'], axis=1, inplace=True)

However, that does not seem very efficient, because it adds and removes columns. It seems that I am making this too complicated, do I overlook a simple more direct solution? Something more like this non-functional code:

incData.loc[incData['y'].isnull(),'y'] = incData[incData['y'].isnull()]['x']*\
                                         fitTable[incData[incData['y'].isnull()]['comp']]['slope']+\
                                         fitTable[incData[incData['y'].isnull()]['comp']]['intercept']

I am pretty new to Pandas, so I sometimes get a bit mixed up with the strict indexing rules...

pandas dataframe python
3 Answers

IIUC:

incData.loc[pd.isna(incData['y']), 'y'] = incData[pd.isna(incData['y'])].apply(lambda row: row['x']*fitTable.loc[row['comp'], 'slope']+fitTable.loc[row['comp'], 'intercept'], axis=1)

incData
comp  x     y
0     A  1   3.0
1     A  2   5.0
2     A  3   7.0
3     B  1   1.0
4     B  2   4.0
5     B  3   7.0
6     B  4  10.0
7     B  5  13.0
8     C  1   4.0
9     C  2   3.0
10    C  3   2.0
11    C  4   1.0
user_102602430

you can use map on the column 'comp' once mask with null value in 'y' like:

mask = incData['y'].isna()
incData.loc[mask, 'y'] = incData.loc[mask, 'x']*\
                         incData.loc[mask,'comp'].map(fitTable['slope']) +\
                         incData.loc[mask,'comp'].map(fitTable['intercept'])

and your non-functional code, I guess it would be something like:

incData.loc[mask,'y'] = incData.loc[mask, 'x']*\
                        fitTable.loc[incData.loc[mask, 'comp'],'slope'].to_numpy()+\
                        fitTable.loc[incData.loc[mask, 'comp'],'intercept'].to_numpy()
user_92747320

merge is another option

# merge two dataframe together on comp
m = incData.merge(fitTable, left_on='comp', right_index=True)
# y = mx+b
m['y'] = m['x']*m['slope']+m['intercept']

   comp  x   y  slope  intercept
0     A  1   3      2          1
1     A  2   5      2          1
2     A  3   7      2          1
3     B  1   1      3         -2
4     B  2   4      3         -2
5     B  3   7      3         -2
6     B  4  10      3         -2
7     B  5  13      3         -2
8     C  1   4     -1          5
9     C  2   3     -1          5
10    C  3   2     -1          5
11    C  4   1     -1          5
user_91778770
Related