What we should do with highly correlated features?

Admin 1
Viewed 5278

In my data set 2 features C1 and C2 are highly correlated. I did following steps. Could you please let me know if it is correct and make sense? do you have a better approach?

First I used linear model to find the fitted line: C1=a*C2+b

from sklearn import linear_model

reg=linear_model.LinearRegression()
y_reg = data1['C1']
x_reg = data1['C2']
reg.fit(x_reg2,y_reg2)
a=reg.coef_
b=reg.intercept_

print(a,b)

After finding a and b I removed C1 and C2 from the dataset and added a new Vriable: new=a*C1+b

My next question is how I can understand if this line is good?

feature-selection correlation linear-regression
2 Answers

In general, it is recommended to avoid having correlated features in your dataset. Indeed, a group of highly correlated features will not bring additional information (or just very few), but will increase the complexity of the algorithm, thus increasing the risk of errors. Depending on the features and the model, correlated features might not always harm the performance of the model but that is a real risk.

You can see this as an interpretation of Occam's razor: with no significant difference in performance, simpler models should be preferred. In your case, a simpler model is a model with only C1 or C2 instead of both, if the performance is similar.

Now what you did when replacing C1 and C2 by a*C1+b actually gets rid of the multicolinearity, but this does not make much sense to me. I don't see any benefits in that compared to keeping C1 only: indeed, you replaced C1 and C2 by a new variable fitted to match... C1! If the linear fit is good, there will be almost no difference.

Feature engineering and feature selection should be justified by underlying theory or at least domain knowledge, so here are several things you could do instead:

  • Apply PCA to the dataset before training the model: this will produce a new and uncorrelated feature set. The disvantage is that you will not be able to explain the model's decision with the original features if that is something you needed.
  • Use a feature selection algorithm. The best algorithm will depend on the data and the model. Here are scikit learn's feature selection algorithms as an example.
  • Simply compare the performance of the model with both C1 and C2, then only C1 and only C2. Then estimate whether the difference in performance makes it worth keeping both features (this is actually applying Occam's razor principle). This can be seen as a "manual" feature selection algorithm.
  • Use domain knowledge to choose which variable to keep: which one is the most relevant to the problrm?. This can be done if the training of the model is too expensive to allow several experiments.

I would also recommend you to read this thread on data science stack exchange as it will give you several other opinions on the issue of multicolinearity. I think this is interesting for you because it will complete the insights I gave here and help you deciding what to do with C1 and C2.

user_70809110

Just remove the feature that are highly correlated. Not removing feature may decrease efficiency and unnecessary cost.

user_148082300
Related