Pandas: How to best select features that are not correlated?

user_85248 0
Viewed 1783

I'm trying to isolate which features of a dataset (i.e. the columns of a Pandas DataFrame) to use for linear regression, and I want to choose those that are not strongly correlated (under the assumption that the independent variables need to be uncorrelated with each other, so we want to remove any which appear to be strongly correlated).

I have isolated the initial list of features that are correlated with my target variable, like so:

# get the absolute correlations for the target variable
correlations_target = abs(df.corr()[target_variable_name])

# filter out those that are below our threshold
correlated_features = correlations_target[correlations_target >= correlation_threshold]

# drop the target variable's column
correlated_features.drop(target_variable_name, inplace=True)

# get the column names for later use
correlated_feature_variable_names = correlated_features.index

I'd now like to go through each of these relevant feature variables and make sure that none of them have strong correlations, and if they do then drop the one with the weakest correlation to the target variable. Here's what I've cooked up for this:

# the collection of feature variable names we'll drop due to their being correlated to other features
correlated_feature_variable_names_to_drop = []

# loop over the feature combinations
for name_1 in correlated_feature_variable_names:
    for name_2 in correlated_feature_variable_names:
        # only look at correlations between separate feature variables
        if name_1 != name_2:
            # drop one of the feature variables if there's a strong correlation
            if abs(df[[name_1, name_2]].corr()[name_1][name_2]) > 0.6:
                # only worry about it if neither of the variables have been added to the drop list
                if (name_1 not in correlated_feature_variable_names_to_drop) and \
                        (name_2 not in correlated_feature_variable_names_to_drop):
                    # drop the one which has the least correlation to the target variable
                    if correlated_features[name_1] >= correlated_features[name_2]:
                        correlated_feature_variable_names_to_drop.append(name_2)
                    else:
                        correlated_feature_variable_names_to_drop.append(name_1)

# drop the variables we've found that qualify
correlated_features.drop(correlated_feature_variable_names_to_drop, inplace=True)

# get the remaining variables' column names for later use
filtered_feature_variable_names = correlated_features.index

The filtered feature set will be used as input for simple regression models. For example:

# fit a simple ordinary least squares model to the features
X = df[filtered_feature_variable_names]
y = df[target_variable_name]
estimate = sm.OLS(y, np.asarray(X)).fit()

# display the regression results
estimate.summary()

Because this is the first time I've tried this I'm unsure if this is the correct way to go about it, and if it is then there might be a more clever or efficient (or "Pythonic") way to perform the filtering than the looping approach I've used above.

pandas linear-regression python
1 Answers

Let me preface this by saying I might be missing part of your intent here because what you've written, while clearly coded, is generic. And I'm having a problem determining if this is the terminology or something else.

I think your intent is to look for and filter out things that are highly correlated to each other but are themselves used to explain the third thing. For example, age and number of days old are really describing the same thing (time). Either, but not both, of which can describe some relation to a third thing like tree diameter. The concept here is Collinearity and you're absolutely right to want to reduce that.

This is where the exploratory data analysis comes in. Some sort of quick visualization like a correlation matrix heatmap would be a good first step.

But, when you're doing regression, you're looking for things that actually are correlated but not collinear with each other. I understand you intend to feed this into a simple regression model but you might consider using sklearn's lasso or ridge regression tools in this case.

There is an additional reason to use sklearn: your example might be different than the one I gave (years correlating with days) which happens to be perfectly correlated. In your case, getting rid of one or the other might be getting rid of some predictive information. Both models will regularize the data, both have the benefit of automated feature selection. Ridge deals with data that have multicollinearity. On the other hand, if you have sparse data, lasso is better.

One last point: while SO is definitely the larger community, I think your question might get a better answer on Stack Exchange's Data Science community.

user_60398300
Related