Detect and Remove Outliers as step of a Pipeline

user_11452928 0
Viewed 1928

I have a problem, I'm trying to build my own class to put into a pipeline in python, but it doesn't work.

The problem I am trying to solve is a multiclass classification problem.

What I want to do this to add a step in the pipeline to detect and remove outliers. I found this detect and remove outliers in pipeline python which is very similar to what I did. This is my class:

from sklearn.neighbors import LocalOutlierFactor
from sklearn.base import BaseEstimator, TransformerMixin
import numpy as np

class OutlierExtraction(BaseEstimator, TransformerMixin):
    def __init__(self, **kwargs ):
        self.kwargs = kwargs

    def transform(self, X, y):
        """
            X should be of shape (n_samples, n_features)
            y should be of shape (n_samples,)
        """

        lof = LocalOutlierFactor(**self.kwargs)
        lof.fit(X)
        nof = lof.negative_outlier_factor_
        return X[nof > np.quantile(nof, 0.95), :], y[nof > np.quantile(nof, 0.95)]

    def fit(self, X, y = None):
        return self

But i get this error in fit_transform return self.fit(X, y, **fit_params).transform(X) TypeError: transform() missing 1 required positional argument: 'y'

The following code is the code i use to call this class:

scaler = preprocessing.RobustScaler()
outlierExtractor = OutlierExtraction()
pca = PCA()
classfier = svm.SVC()

pipeline = [('scaler', scaler),
            ('outliers', outlierExtractor),
        ('reduce_dim', pca),
        ('classfier', classfier)]

pipe = Pipeline(pipeline)

params = {
    'reduce_dim__n_components': [5, 15],
    'classfier__kernel': ['rbf'],
    'classfier__gamma': [0.1],
    'classfier__C': [1],
    'classfier__decision_function_shape':['ovo']}

my_scoring = 'f1_macro'
n_folds = 5
gscv = GridSearchCV(pipe, param_grid=params, scoring=my_scoring, n_jobs=-1, cv=n_folds, refit=True)
gscv.fit(train_x, train_y)
scikit-learn classification regression python
2 Answers

@TimCroydon has it right: sklearn currently assumes that transformers only transform their independent variables. There have been some long discussions on how best to relax this:

The scikit-learn-contrib package imbalanced-learn supports a number of resamplers, which have similar effect but different context; you may be able to use that, but perhaps it will look a little weird to be fit_sampleing when removing outliers. Anyway, they have a custom version of Pipeline that deals with that resampling elegantly.

Finally, you may be able to just override the fit_transform method in your custom class. It seems like it should work for this situation, though it may cause issues elsewhere.

user_104958930

The error is because the transform method def transform(self, X, y) requires both X and y to be passed in, but whatever is calling it is only passing X. (I can't see where it's called from in your code so assume it's being called by the underlying library).

I don't know if making y optional (def transform(self, X, y=None) and modifying your method would work in this case. Otherwise, you'll have to figure out how to get the calling code to pass y, or provide it another way.

I'm not familiar with the library, but looking at the source code shows that transform() should only take a single parameter X:

        if y is None:
            # fit method of arity 1 (unsupervised transformation)
            return self.fit(X, **fit_params).transform(X)
        else:
            # fit method of arity 2 (supervised transformation)
            return self.fit(X, y, **fit_params).transform(X)
user_1648610
Related