On what data does sklearn Transformation operate?

user_622097 0
Viewed 415

I am writing a set of custom transformations in sklearn in order to clean the data in a pipeline. Each custom transformation takes two Pandas DataFrame as parameters for fit and transform, transform returns two DataFrames as well (see examples below). This works fine when there is only one Transformer in a pipeline: DataFrames in and DataFrames out.

However, when both Rransformers are combined in a Pipeline, like this:

pipeline = Pipeline ([
        ('remove_missing_columns', RemoveAllMissing (['mailing_address_str_number'])),
        ('remove_rows_based_on_target', RemoveMissingRowsBasedOnTarget ()),
        ])

X, y = pipeline.fit_transform (X, y)

==>TypeError: tuple indices must be integers or slices, not Series

class RemoveMissingRowsBasedOnTarget mysteriously receives a tuple as input. When I switch the positions of the Transformers like this

pipeline = Pipeline ([
        ('remove_rows_based_on_target', RemoveMissingRowsBasedOnTarget ()),
        ('remove_missing_columns', RemoveAllMissing (['mailing_address_str_number'])),
        ])

==> AttributeError: 'tuple' object has no attribute 'apply'

The error occurs in the class RemoveAllMissing. In both cases the error message is indicated with ==> above the line where the error occurs. I think I did quite some reading on what exactly what exactly could be going on, but I couldn't find anything on this subject. Could somebody tell me what I'm doing wrong? Below you'll find the code for the problem isolated.

import numpy as np
import pandas as pd
import random
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.pipeline import Pipeline

def create_data (rows, cols, frac_nan, random_state=42):
    random.seed (random_state)
    X = pd.DataFrame (np.zeros ((rows, cols)), 
                      columns=['col' + str(i) for i in range (cols)], 
                      index=None)
    # Create dataframe of (rows * cols) with random floating points
    y = pd.DataFrame (np.zeros ((rows,)))
    for row in range(rows):
        for col in range(cols):
            X.iloc [row,col] = random.random()
        X.iloc [row,1] = np.nan # column 1 exists colely of NaN's
        y.iloc [row] = random.randint (0, 1)
    # Assign NaN's to a fraction of X
    n = int(frac_nan * rows * cols)
    for i in range (n):
        row = random.randint (0, rows-1)
        col = random.randint (0, cols-1)
        X.iloc [row, col] = np.nan
    # Same applies to y
    n = int(frac_nan * rows)
    for i in range (n):
        row = random.randint (0, rows-1)
        y.iloc [row,] = np.nan

    return X, y    

class RemoveAllMissing (BaseEstimator, TransformerMixin):
    # remove columns containg NaN only
    def __init__ (self, requested_cols=[]):
        self.all_missing_data = requested_cols

    def fit (self, X, y=None):
        # find empty columns == columns with all missing data
        missing_cols = X.apply (lambda x: x.count (), axis=0)
        for idx in missing_cols.index:
            if missing_cols [idx] == 0:
                self.all_missing_data.append (idx)

        return self

    def transform (self, X, y=None):
        print (">RemoveAllMissing - X shape: " + str (X.shape), " y shape: " + str (y.shape), 'type (X):', type(X))
        for all_missing_predictor in self.all_missing_data:
            del X [all_missing_predictor]

        print ("<RemoveAllMissing - X shape: " + str (X.shape), " y shape: " + str (y.shape), 'type (X):', type(X))
        return X, y

    def fit_transform (self, X, y=None):
        return self.fit (X, y).transform (X, y)

class RemoveMissingRowsBasedOnTarget (BaseEstimator, TransformerMixin):
    # remove each row where target contains one or more NaN's
    def __init__ (self):
        self.missing_rows = []

    def fit (self, X, y = None):
        # remove all rows where the target value is missing data
        print (type (X))
        if y is None:
            print ('RemoveMissingRowsBasedOnTarget: target (y) cannot be None')
            return self

        self.missing_rows = np.array (y.notnull ()) #  false = missing data

        return self

    def transform (self, X, y=None):
        print (">RemoveMissingRowsBasedOnTarget - X shape: " + str (X.shape), " y shape: " + str (y.shape), 'type (X):', type(X))
        if y is None:
            print ('RemoveMissingRowsBasedOnTarget: target (y) cannot be None')
            return X, y

        X = X [self.missing_rows].reset_index ()
        del X ['index']
        y = y [self.missing_rows].reset_index ()
        del y ['index']  

        print ("<RemoveMissingRowsBasedOnTarget - X shape: " + str (X.shape), " y shape: " + str (y.shape), 'type (X):', type(X))
        return X, y

    def fit_transform (self, X, y=None):
        return self.fit (X, y).transform (X, y)

pipeline = Pipeline ([
        ('RemoveAllMissing', RemoveAllMissing ()),
        ('RemoveMissingRowsBasedOnTarget', RemoveMissingRowsBasedOnTarget ()),
        ])

X, y = create_data (25, 10, 0.1)
print ("X shape: " + str (X.shape), " y shape: " + str (y.shape), 'type (X):', type(X))
X, y = pipeline.fit_transform (X, y) 
#X, y = RemoveAllMissing ().fit_transform (X, y)
#X, y = RemoveMissingRowsBasedOnTarget ().fit_transform (X, y)

Edit As @Vivek requested I have replaced the original code by code where the problem is isolated and runs stand alone. The code as is will crash somewhere because a tuple is transferred as parameter instead of a DataFrame. Pipeline changes the datatypes and I cannot find that in the documentation. When one comments out the call to the pipeline and removes the comments before the separate calls of the transformer everyting works fine, like this:

#X, y = pipeline.fit_transform (X, y) 
X, y = RemoveAllMissing ().fit_transform (X, y)
X, y = RemoveMissingRowsBasedOnTarget ().fit_transform (X, y)
scikit-learn pandas python
1 Answers

Ok, now I have got the error, which seems to be with your classes returning both X, y whereas a pipeline can take input of y (and pass it along its internal transformers), it assumes that y is constant throughout and never returned by any transform() method. Which is not the case in your code. If you can separate that part elsewhere, it can work.

See this line in the source code of pipeline:

    if hasattr(transformer, 'fit_transform'):
        res = transformer.fit_transform(X, y, **fit_params)
    else:
        res = transformer.fit(X, y, **fit_params).transform(X)

You are returning two values (X,y) but its contained in only a single variable res and hence it becomes a tuple. Which then fails in your next transformer.

You can handle such data by unzipping the tuple into X, y like this:

class RemoveMissingRowsBasedOnTarget (BaseEstimator, TransformerMixin):
    ...
    ...

    def fit (self, X, y = None):
        # remove all rows where the target value is missing data
        print (type (X))
        if isinstance(X, tuple):
            y=X[1]
            X=X[0]

        ...
        ...

        return self

    def transform (self, X, y=None):
        if isinstance(X, tuple):
            y=X[1]
            X=X[0]

        ...
        ...

        return X, y

    def fit_transform(self, X, y=None):
        self.fit(X, y).transform(X, y)

Make sure you do this for all subsequent transformers in the pipeline. But I would recommend you to separate the X and y processing. Also, I found that there are some relevant issues for transforming the target variables y inside the pipeline which you can have a look at:

user_33749960
Related