Remove features based on correlation in case of mixed continuous & categorical features

Viewed 17

I am working on a machine learning regression task with mixed continuous and categorical features in Python .

I apply one-hot encoding on categorical features as can be seen below:

from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split
from sklearn.impute import SimpleImputer
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, MinMaxScaler


# -----------------------------------------------------------------------------
# Data
# -----------------------------------------------------------------------------

# Ames 
X, y = fetch_openml(name="house_prices", as_frame=True, return_X_y=True)

# In this dataset, categorical features have "object" or "non-numerical" data-type. 
numerical_features = X.select_dtypes(include='number').columns.tolist()   # 37
categorical_features = X.select_dtypes(include='object').columns.tolist()   # 43


X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.3, random_state=0)


# -----------------------------------------------------------------------------
# Data preprocessing
# -----------------------------------------------------------------------------

numerical_preprocessor = Pipeline(steps=[
    ('impute', SimpleImputer(strategy='mean')),
    ('scale', MinMaxScaler())
])


categorical_preprocessor = Pipeline(steps=[
    ('impute', SimpleImputer(strategy='most_frequent')),
    ('one-hot', OneHotEncoder(handle_unknown='ignore', sparse=False))
])
   

preprocessor = ColumnTransformer(transformers=[
    ('number', numerical_preprocessor, numerical_features),
    ('category', categorical_preprocessor, categorical_features)
], 
        verbose_feature_names_out=True,   
)


X_train_processed = preprocessor.fit_transform(X_train)  
X_test_processed = preprocessor.transform(X_test) 

I want to remove highly correlated features by the following algorithm:

  • Find Pearson correlation coefficient between all features.
  • If correlation > threshold:
    • Drop one of the features which has lower correlation with objective variable (which is a continuous variable)

However, I am not sure which method is suitable to calculate correlation between :

  • continuous features & one-hot encoded categorical features
  • one-hot encoded categorical features & continuous objective variable

Any advice is appreciated.


Assume that the machine learning task is a classification task. Which method do you recommend to calculate correlation between :

  • one-hot encoded categorical features & categorical objective variable
  • continuous features & categorical objective variable
0 Answers
Related