sklearn ColumnTransformer: Duplicate columns in transformers

user_12448301 0
Viewed 873

I am looking for a help building a data preprocessing pipleline using sklearn's ColumnTransformer functions.

Currently my pipleline looks something like this:

from scipy.stats.mstats import winsorize 
from sklearn.compose import ColumnTransformer 
from sklearn.impute import SimpleImputer

ColumnTransformer(remainder='passthrough',
                  transformers=[
                              ('Winsorize', FunctionTransformer(winsorize,
                                                     kw_args={'axis': 0, 'inplace': False, 'limits': [0, 0.01]}), ['feat_1','feat_2']),
                                ('num_impute', SimpleImputer(strategy='median'), ['feat_3', 'feat_4']),
                                ])

Note that each transformer is provided a unique set of features.

The issue I am encountering is how to apply stacked analysis for the same features. For example,

ColumnTransformer(remainder='passthrough',
                  transformers=[
                              ('Winsorize', FunctionTransformer(winsorize,
                                                     kw_args={'axis': 0, 'inplace': False, 'limits': [0, 0.01]}), ['feat_1','feat_2']),
                                ('num_impute', SimpleImputer(strategy='median'), ['feat_1', 'feat_2', 'feat_3']),
                                ])

Note that feat_1 and feat_2 where provided for both transformers.

a pipeline like this will create duplicate columns for feat_1 and feat_2 (Two columns based on Winsorize, and two columns based on num_impute)

scikit-learn pipeline python
2 Answers

From what I understand, you can use two ColumnTransformer and one FeatureUnion to achieve what you want. One ColumnTransformer will have remainder='passthrough' to keep all columns except the ones being transformed, and the other will have remainder='drop'. That could look like this:

from sklearn.pipeline import FeatureUnion

ct1 = ColumnTransformer(
    remainder='passthrough',
    transformers=[(
        'Winsorize', FunctionTransformer(
            winsorize,
            kw_args={'axis': 0, 'inplace': False, 'limits': [0, 0.01]}
        ), 
        ['feat_1', 'feat_2']
    )]
)

ct2 = ColumnTransformer(
    remainder='drop',
    transformers=[('num_impute', SimpleImputer(strategy='median'), ['feat_1', 'feat_2', 'feat_3'])],
)

union = FeatureUnion([('ct1', ct1), ('ct2', ct2)])
user_82537690

You can use a Pipeline as the transformer for each column set if you need multiple transformations applied to a column.

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer

imputer = SimpleImputer(strategy="median")
winsorize = FunctionTransformer(
  winsorize, kw_args={"axis": 0, "inplace": False, "limits": [0, 0.01]})

ColumnTransformer([
  ("F1,F2", Pipeline([
    ("Winsorize", winsorize),
    ("Impute", imputer)
  ]), ["feat_1","feat_2"]),
  ("F3", imputer, ["feat_3"])
], remainder="passthrough")
user_4963510
Related