Wrapping similar transformers with Pipeline and Column Transformer gives different results

user_2280637 0
Viewed 117

I have the following toy dataset.

import random
import numpy as np
import pandas as pd
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.base import TransformerMixin, BaseEstimator
from sklearn.pipeline import Pipeline
from sklearn.compose import make_column_selector as selector
K = 6
tar = random.choices([1, 0], weights=[0.2, 0.8], k=K)
cat = random.choices(["A", "B", "C"], k=K)
dog = random.choices(["D", "E", "F"], k=K)
n1 = [random.randint(1,9) for _ in range(K)]
n2 = [random.randint(1,9) for _ in range(K)]
df = pd.DataFrame.from_dict({"tar": tar, "cat": cat, 'dog': dog, 'n1': n1})

I run the following code:

class CategoricalFeaturesSelector(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        return self
    def transform(self, X):
        return X.loc[:, X.select_dtypes(include=['int64', 'float64']).columns.tolist()]


tr3 = Pipeline([
    ('selector3', CategoricalFeaturesSelector())
])
tr3.fit_transform(df)

which returns the output in a DataFrame format as I expected.

enter image description here

But when I revise the code a little bit as what follows. It returns a numpy ndarray instead of a Pandas DataFrame object, even though I have explicitly specified it in the definition of transform.

from sklearn.compose import make_column_selector as selector
class dummyTransformer(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        return self
    def transform(self, X):
        return pd.DataFrame(X, columns=selector(dtype_include='object')(X)) 


tr4 = ColumnTransformer(transformers=[
    ('dum', dummyTransformer(), selector(dtype_include='object'))
])
tr4.fit_transform(df)

enter image description here

Can anyone tell me what causes the difference? I want to have it return a DataFrame object rather than a numpy ndarray.

scikit-learn pandas numpy python
1 Answers

Pipeline will return the result from the transform() method of CategoricalFeaturesSelector

return last_step.fit(Xt, y,**fit_params_last_step).transform(Xt)

while ColumnTransformer horizontally stacks the results from the transformers. You can verify this on the source code

return self._hstack(list(Xs))

which calls np.stack() on the input parameter list(Xs)

user_124833460
Related