How to reduce memory usage in Python while doing Variance Inflation Factor (VIF)

Viewed 15

I'm doing the following code below in an attempt to make the Variance Inflation Factor of my variables. However, the amount of memory used has been outrageous, as I have almost 10 million rows of data.

Any suggestions on how to make such code more efficient and less memory consuming?

VIF_X = VIF_X.drop(['count'],axis=1)
VIF_X = VIF_X[list(VIF_X.columns)]

vif_info = pd.DataFrame()
vif_info['VIF'] = [variance_inflation_factor(VIF_X.values, i) for i in range(VIF_X.shape[1])]
vif_info['Column'] = VIF_X.columns
vif_info.sort_values('VIF', ascending=False)

The code takes a lot of time during vif_info = pd.DataFrame(), so I think this procedure is consuming the most memory.

0 Answers
Related