Pandas groupby transform covariance

user_8326436 0
Viewed 348

My DataFrame:

A = pd.DataFrame({'group':['A','A','A','A','B','B','B'],
              'value1':[1,2,3,4,5,6,7],
              'value2':[8,5,4,3,7,8,8]})

I'd like to have a new column, covariance, that show the covariance between [value1,value2] on the group-level, for each observation.

Therefore I tried:

A['covariance'] = A.groupby('group')[['value1','value2']].transform(lambda x: x['value1'].cov(x['value2']),axis=1)

But this results in error:

TypeError: Transform function invalid for data types

What am I missing?

pandas
1 Answers

I believe you need to specify the columns a bit differently:

>>> A.groupby('group')['value1'].transform(lambda x: x.cov(A['value2']))
0   -2.666667
1   -2.666667
2   -2.666667
3   -2.666667
4    0.500000
5    0.500000
6    0.500000

This works like:

>>> for _, frame in A.groupby('group'):
...     print(frame['value1'].cov(frame['value2']))
... 
-2.6666666666666665
0.5

Side note - I find this syntax a bit confusing because A.groupby('group') is an iterator over tuples of ('group', 'df'). But I think this stems from the fact that

The transform function must: Operate column-by-column on the group chunk. The transform is applied to the first group chunk using chunk.apply. (source)

And as a disclosure, generally, throwing a lambda inside the call may (may, but not always) make things slower by putting computation in the Python space (rather than Cython).

user_79545040
Related