Target encoding using nested CV in sklearn pipeline

Viewed 75

I am doing target encoding for my column, using nested cross validation approach (to avoid leakage) as mentioned here, here and here.

If I had to include my target encoding (by a custom transformer), in the sklearn pipeline, I need different transform function from the train set and the test set. This is because, for the train folds, the encoding is calculated using a further kfold split of the train data. Whereas for the test fold, encoding is mean of the train.

I know the sklearn pipeline will apply the same transformation for train and test split in the cv, is there a way to apply separate transformations for train and test splits using a sklearn pipeline and custom transformer

1 Answers

The category_encoders package implements some target encoders. It gets around the issue of different training and testing dataset behavior by implementing a fit_transform method that is not equivalent to just fit followed by transform: fit_transform performs the training-set transformation, while transform performs the test/production set transformation.

Related