How to find customized average which is based on weightage including handling of nan value in pandas?

user_10646971 0
Viewed 53

I have a data frame df_ss_g as

ent_id,WA,WB,WC,WD
123,0.045251836,0.614582906,0.225930615,0.559766482
124,0.722324239,0.057781167,,0.123603561
125,,0.361074325,0.768542766,0.080434134
126,0.085781742,0.698045853,0.763116684,0.029084545
127,0.909758657,,0.760993759,0.998406211
128,,0.32961283,,0.90038336
129,0.714585519,,0.671905291,
130,0.151888772,0.279261613,0.641133263,0.188231227

now I have to compute the average(AVG_WEIGHTAGE) which is based on a weightage i.e. =(WA*0.5+WB*1+WC*0.5+WD*1)/(0.5+1+0.5+1)

but while I am computing it using below method i.e.

df_ss_g['AVG_WEIGHTAGE']= df_ss_g.apply(lambda x:((x['WA']*0.5)+(x['WB']*1)+(x['WC']*0.5)+(x['WD']*1))/(0.5+1+0.5+1) , axis=1)

IT output as i.e. for NaN value it is giving NaN as AVG_WEIGHTAGE as null which is wrong. enter image description here

all I wanted is that null should not be considered in denominator and numerator e.g.

ent_id,WA,WB,WC,WD,AVG_WEIGHTAGE
128,,0.32961283,,0.90038336,0.614998095   i.e. (WB*1+WD*1)/1+1
129,0.714585519,,0.671905291,,0.693245405 i.e. (WA*0.5+WC*0.5)/0.5+0.5
data-science pandas dataframe numpy python
3 Answers

IIUC:

import numpy as np

weights = np.array([0.5, 1, 0.5, 1]))
values = df.drop('ent_id', axis=1)

df['AVG_WEIGHTAGE'] = np.dot(values.fillna(0).to_numpy(), weights)/np.dot(values.notna().to_numpy(), weights)


df['AVG_WEIGHTAGE']
0    0.436647
1    0.217019
2    0.330312
3    0.383860
4    0.916891
5    0.614998
6    0.693245
7    0.288001
user_102602430

Try this method using dot products -

def av(t):
    #Define weights
    wt = [0.5, 1, 0.5, 1]
    
    #Create a vector with 0 for null and 1 for non null
    nulls = [int(i) for i in ~t.isna()]
    
    #Take elementwise products of the nulls vector with both weights and t.fillna(0)
    wt_new = np.dot(nulls, wt)
    t_new = np.dot(nulls, t.fillna(0))
    
    #return division
    return np.divide(t_new,wt_new)

df['WEIGHTED AVG'] = df.apply(av, axis=1)
df = df.reset_index()
print(df)

   ent_id        WA        WB        WC        WD  WEIGHTED AVG
0     123  0.045252  0.614583  0.225931  0.559766      0.481844
1     124  0.722324  0.057781       NaN  0.123604      0.361484
2     125       NaN  0.361074  0.768543  0.080434      0.484020
3     126  0.085782  0.698046  0.763117  0.029085      0.525343
4     127  0.909759       NaN  0.760994  0.998406      1.334579
5     128       NaN  0.329613       NaN  0.900383      0.614998
6     129  0.714586       NaN  0.671905       NaN      1.386491
7     130  0.151889  0.279262  0.641133  0.188231      0.420172
user_47559540

It boils down to masking the nan values with 0 so they don't contribute to either weights or sum:

# this is the weights
weights = np.array([0.5,1,0.5,1])

# the columns of interest
s = df.iloc[:,1:]

# where the valid values are
mask = s.notnull()

# use `fillna` and then `@` for matrix multiplication
df['AVG_WEIGHTAGE'] = (s.fillna(0) @ weights) / (mask@weights)
user_42384080
Related