Use sklearn transformers on list of columns and preserve the input columns

user_7372226 0
Viewed 269

Using sklearn transformers, I'd like to be able to apply transformations to a list of columns and have the transformer create new columns with the transformed values rather than apply them to the existing ones. Is this possible? The transformer also needs to slot into a a Pipeline.

My goal is to compare the original columns and transformed columns. A wrapper class around the transformer could work, but I wonder if there's an easier way? Thank you.

scikit-learn python
2 Answers

You could use FeatureUnion and concatenate your transformer with an identity transformer which leaves your columns untouched.

from sklearn.pipeline import FeatureUnion
from sklearn.preprocessing import FunctionTransformer
import numpy as np

union = FeatureUnion([("id", FunctionTransformer(lambda x: x)),
                      ("add_one", FunctionTransformer(lambda x: x + 1))])

X = np.array([[0, 1, 2], [3, 4, 5]])
union.fit_transform(X)

>>> array([[0, 1, 2, 1, 2, 3], [3, 4, 5, 4, 5, 6]])

There seems to be no IdentityTransformer class in sklearn right now. I used a FunctionTransformer with the identity function, but I don't know if there is any smarter way of doing this.

user_120806440

The easiest way to do this would be to use a function with an argument that accepts a list of the features you would like to transform. From there you have 2 options:

  1. This is the method you requested. Since the function transforms only those features that you requested, you can replace the values of the features you would like to have transformed within the function.

  2. This is what I would recommend. Create a copy of the original dataframe and "paste" all the transformed features into it. You can then print the two dataframes in different cells (I'm assuming you're using jupyter notebooks) to compare the differences.

This would be the function to use:

    def transform_data(scaler, df, feats_to_transform):
        features = scaled_df[feats_to_transform]
        transformed_feats = scaler.fit_transform(features.values) # The transformers take only 2d arrays
                      ​
        return transformed_feats

Method 1:

df  = pd.read_csv('path/to/csv')
scaler = StandardScaler() # from sklearn.preprocessing import StandardScaler

feats_to_transform = ['feat1, feat2, feat3'] 
transformed_feats = transform_data(scaler, df, feats_to_transform)

df[feats_to_transform] = transformed_feats

Method 2:

df  = pd.read_csv('path/to/csv')
scaled_df = df.copy(deep=True) # Using deep copy prevents alteration of original data
scaler = StandardScaler() # from sklearn.preprocessing import StandardScaler

feats_to_transform = ['feat1, feat2, feat3'] 
transformed_feats = transform_data(scaler, df, feats_to_transform)

scaled_df [feats_to_transform] = transformed_feats

# now compare in different cells
df.head()
scaled_df.head()
user_134635740
Related