fit vs fit_transform in pipeline

user_1497720 0
Viewed 2097

In this page https://www.kaggle.com/baghern/a-deep-dive-into-sklearn-pipelines

It calls fit_transfrom for tranforming the data as follows:

from sklearn.pipeline import FeatureUnion

feats = FeatureUnion([('text', text), 
                      ('length', length),
                      ('words', words),
                      ('words_not_stopword', words_not_stopword),
                      ('avg_word_length', avg_word_length),
                      ('commas', commas)])

feature_processing = Pipeline([('feats', feats)])
feature_processing.fit_transform(X_train)

While during training with feature processing, it only uses fit then predict

from sklearn.ensemble import RandomForestClassifier

pipeline = Pipeline([
    ('features',feats),
    ('classifier', RandomForestClassifier(random_state = 42)),
])

pipeline.fit(X_train, y_train)

preds = pipeline.predict(X_test)
np.mean(preds == y_test)

The question is, is the fit doing the transformation on X_train (as what is achieved by transform, since we are not calling fit_transform here) for second case?

scikit-learn pipeline python-3.x
1 Answers

sklearn-pipeline has some nice features. It perform several task in a very clean way. We define our features, its transformation and list of classifiers, we want to perform, all in one function.

In the first step of this

pipeline = Pipeline([
    ('features',feats),
    ('classifier', RandomForestClassifier(random_state = 42)),
])

you have defined the features's name and its transformation function(that is incorporated in feat), in second step, you have defined the classifier's name and classifier classifier.

Now while calling pipeline.fit, it first fit features and transform it, then fit the classifier on the transformed features. So, it does some steps for us. More you can check-here

user_92772450
Related