Known mathemathical formula states Var(X) = Var(E(X|Z)) + E(Var(x|Z)) I followed the formula in pandas, as
df = pd.DataFrame({'X': [2,2.1,1.9,2.2,2,10,9.5,11],
'Z': [1,1,1,1,1,0,0,0]})
explained = df.groupby('Z').mean().var()
residual = df.groupby('Z').var().mean()
variance = df[['X']].var()
print(f'variance: {variance[0]}, explained + residual: {explained[0] + residual[0]}')
and got the result
variance: 17.86410714285714, explained + residual: 33.31952222222222
Why is that? My guess is my formula for explained and residual variance is not correct. Should I use the frequency of each class in Z to weight the mean and variance?