standardize pandas groupby results

Rense Lange 0
Viewed 31

I am using pandas to get subgroup averages, and the basics work fine. For instance,

d = np.array([[1,4],[1,1],[0,1],[1,1]])
m = d.mean(axis=1)

p = pd.DataFrame(m,index='A1,A2,B1,B2'.split(','),columns=['Obs'])
pprint(p)

x = p.groupby([v[0] for v in p.index])
pprint(x.mean('Obs'))

x = p.groupby([v[1] for v in p.index])
pprint(x.mean('Obs'))

YIELDS:

    Obs
A1  2.5
A2  1.0
B1  0.5
B2  1.0

    Obs
A  1.75. <<<< 1.75 is (2.5 + 1.0) / 2
B  0.75

   Obs
1  1.5
2  1.0

But, I also need to know how much A and B (1 and 2) deviate from their common mean. That is, I'd like to have tables like:

    Obs   Dev
A  1.75  0.50  <<< deviation of the Obs average, i.e., 1.75 - 1.25
B  0.75 -0.50  <<< 0.75 - 1.25 = -0.50

   Obs    Dev
1  1.5   0.25
2  1.0  -0.25

I can do this using loc, apply etc - but this seems silly. Can anyone think of an elegant way to do this using groupby or something similar?

group-by pandas python
1 Answers

Aggregate the means, then compute the difference to the mean of means:

(p.groupby(p.index.str[0])
  .agg(Obs=('Obs', 'mean'))
  .assign(Dev=lambda d: d['Obs']-d['Obs'].mean())
)

Or, in case of a variable number of items if you want the difference to the overall mean (not the mean of means!):

(p.groupby(p.index.str[0])
  .agg(Obs=('Obs', 'mean'))
  .assign(Dev=lambda d: d['Obs']-p['Obs'].mean()) # notice the p (not d)
)

output:

    Obs  Dev
A  1.75  0.5
B  0.75 -0.5
mozway0
Related