sklearn pipeline not remembering its state when applied on the testing set

user_8818287 0
Viewed 93

When calling the fit_transform method of my pipeline on the training set and then the transform method on the testing set, the pipeline is updating its internal state according to the testing set, instead of simply applying what was learnt on the training set.

E.g. given the following dataframe:

df = pd.DataFrame({
    'Sex':['female', 'male', 'male', 'male', 'female', 'female','neutral', 'male'],
    'Survived':['no', 'no', 'yes', 'no', 'yes', 'no', 'yes', 'no']
})

And having it split into X_train, X_test, y_train, y_test:
Important: Please note that I'm splitting in a way that only the test set has the value "neutral". 

X_train = df.loc[:4,'Sex']
y_train = df.loc[:4,'Survived']

X_test = df.loc[5:, 'Sex']
y_test = df.loc[5:, 'Survived']

Below I created a transformer called Dummifier and inserted in my pipeline (the pipeline has here only one method for the sake of simplicity):

class Dummifier(BaseEstimator, TransformerMixin):

    def fit(self, X, y=None):
        return self

    def transform(self, X, y=None):
        X_dummies = pd.get_dummies(X)

        return X_dummies


my_pipe = Pipeline([
    ('get_dummies', Dummifier())
])

Now, when I called the fit_transform in my training set and then the transform method on the testing set, the given output is the below:

X_train_trans = my_pipe.fit_transform(X_train, y_train)
X_test_trans = my_pipe.transform(X_test)

Output:
    female male neutral
5   1      0    0
6   0      0    1
7   0      1    0

Question: Since there was no value "neutral" in the training set, why is the transformer creating now a column "neutral" in the testing set?

The expected output is:

Output:
    female male 
5   1      0    
6   0      0    
7   0      1

I've already try to use the OneHotEncoder from sklearn but the output is basically the same.

scikit-learn machine-learning python
2 Answers

OneHotEncoder's handle_unknown param needs to be set as ignore for your requirement. May be this can help!

from sklearn.preprocessing import OneHotEncoder
from sklearn.pipeline import Pipeline
import pandas as pd


df = pd.DataFrame({
    'Sex': ['female', 'male', 'male', 'male', 'female',
            'female', 'neutral', 'male'],
    'Survived': ['no', 'no', 'yes', 'no', 'yes', 'no', 'yes', 'no']
})


X_train = df.loc[:4, 'Sex'].to_frame()
y_train = df.loc[:4, 'Survived']

X_test = df.loc[5:, 'Sex'].to_frame()
y_test = df.loc[5:, 'Survived']


my_pipe = Pipeline([
    ('get_dummies', OneHotEncoder(handle_unknown='ignore'))
])

my_pipe.fit_transform(X_train)

print(my_pipe.transform(X_test).toarray())


# [[1. 0.]
#  [0. 0.]
#  [0. 1.]]
user_63476290

Following your code as-is, your two outputs for the transforms are as follows:

X_train_trans

    female  male
0     1      0
1     0      1
2     0      1
3     0      1
4     1      0

X_test_trans

    female  male  neutral
5     1      0       0
6     0      0       1
7     0      1       0

Your question is: why is the transformer creating now a column "neutral" in the testing set? It looks like the reason is because you declare the X_test_trans set to be X_test_trans = my_pipe.transform(X_test), which takes in the X_test data, which is as follows:

X_test

5     female
6    neutral
7       male

The code is doing exactly what you are telling it to do. So let's think of a solution:

from sklearn.preprocessing import OneHotEncoder

df = pd.DataFrame({
    'Sex':['female', 'male', 'male', 'male', 'female', 'female','neutral', 'male'],
    'Survived':['no', 'no', 'yes', 'no', 'yes', 'no', 'yes', 'no']
})

features = pd.DataFrame(OneHotEncoder().fit_transform(df['Sex'].values.reshape(-1, 1)).toarray())

The one-hot encoder will take your three categories and convert them into a format the learning algo will recognise. After this stage, you can then begin to split your data into test and train using train_test_split:

from sklearn.model_selection import train_test_split

features = pd.DataFrame(OneHotEncoder().fit_transform(df['Sex'].values.reshape(-1, 1)).toarray())
labels = df['Survived']

X_train, X_test, y_train, y_test = train_test_split(features, labels)
user_87630970
Related