I'm doing the following code below in an attempt to make the Variance Inflation Factor of my variables. However, the amount of memory used has been outrageous, as I have almost 10 million rows of data.
Any suggestions on how to make such code more efficient and less memory consuming?
VIF_X = VIF_X.drop(['count'],axis=1)
VIF_X = VIF_X[list(VIF_X.columns)]
vif_info = pd.DataFrame()
vif_info['VIF'] = [variance_inflation_factor(VIF_X.values, i) for i in range(VIF_X.shape[1])]
vif_info['Column'] = VIF_X.columns
vif_info.sort_values('VIF', ascending=False)
The code takes a lot of time during vif_info = pd.DataFrame(), so I think this procedure is consuming the most memory.