I hope you are well.
I'm currently trying to calculate regression on some group of a data frame but without success. I'm succeeding to calculate what I want but really don't know how to reintegrate the result to my origin dataframe due to the out data structure. I try 2 functions.
I succeed for quintile and give you the code.
Sorry for the size of this message but I'm trying to be the clearest I can.
Package
import pandas as pd
from collections import OrderedDict
import statsmodels.api as sm
import numpy as np
from sklearn.linear_model import LinearRegression
Functions
def regress(data, yvar, xvars):
Y = data[yvar]
X = data[xvars]
X['intercept'] = 1.
result = sm.OLS(Y, X).fit()
y_pred = result.predict()
residual = Y - y_pred
return residual
def Reg_func(x,y):
# Cross Sectional Regression
x = np.array(x).reshape((-1,1))
y = np.array(y)
model = LinearRegression().fit(x, y)
y_pred = model.intercept_ + np.sum(model.coef_ * x,axis=1)
residual = y - y_pred
return residual
Dataframe Creation
ind = ['I1', 'I2', 'I3', 'I4', 'I5', 'I6', 'I7', 'I8', 'I9', 'I10', 'I11', 'I12', 'I13', 'I14', 'I15', 'I16', 'I17', 'I18', 'I19', 'I20']
Axe = ['A', 'A', 'B', 'A', 'A', 'A', 'A', 'B', 'A', 'A', 'A', 'B', 'B', 'A', 'B', 'B', 'B', 'B', 'B', 'B']
df = pd.DataFrame(np.random.randn(20, 2), index = ind, columns=['C1', 'C2'])
df.insert(0,'Axe',Axe)
If you know a better way to create it I would be greatfull :).
Calculations
# Quintile groupé par Axe
QC1 = df.groupby(['Axe'])['C1'].apply(lambda x: pd.qcut(x, 5, labels=False)+1)
print(QC1)
QC1 respect the df structure then it's easy to integrate the result to df
# Simple regression without groupby
res_reg = Reg_func(newdf['C1'], newdf['C2'])
Res_REg is ok with df structure
# Regression per group with Reg_func fucntion
res_reg_group = (df.groupby('Axe').apply(lambda x: Reg_func(x['C1'], x['C2'])))
print(res_reg_group)
I really don't know how to reintegrate the result to df due to it structure
# Regression per group with regress function
res_reg_group2 = df.groupby('Axe').apply(regress, 'C1', ['C2'])
print(res_reg_group2)
The res_reg_group2 seem to have a better structure (keep index) but not sure to know how to combine it with my df dataframe. Moreover this function regress doesn't work for a simple regressions (without groupby).
Thanks for your help and take care