Combine rows in scikit during ColumnTransformer transform

Question

As a working example modified from here, say I have

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.preprocessing import OneHotEncoder

X = pd.DataFrame(
    {'type': ['type1', 'type1', 'type1', 'type1', 'type2', 'type2', 'type2', 'type3', 'type3', 'type4'],
     'feat': ['A','B','C','D','A','B','C','A','B','A']})

column_trans = ColumnTransformer(
    [('type_', OneHotEncoder(),['type']),
     ('feate_', OneHotEncoder(),['feat'])],
    remainder='drop')

column_trans.fit(X)
column_trans.get_feature_names()
column_trans.transform(X).toarray()

which will output

array([[1., 0., 0., 0., 1., 0., 0., 0.],
       [1., 0., 0., 0., 0., 1., 0., 0.],
       [1., 0., 0., 0., 0., 0., 1., 0.],
       [1., 0., 0., 0., 0., 0., 0., 1.],
       [0., 1., 0., 0., 1., 0., 0., 0.],
       [0., 1., 0., 0., 0., 1., 0., 0.],
       [0., 1., 0., 0., 0., 0., 1., 0.],
       [0., 0., 1., 0., 1., 0., 0., 0.],
       [0., 0., 1., 0., 0., 1., 0., 0.],
       [0., 0., 0., 1., 1., 0., 0., 0.]])

What I'm trying to get is instead an encoding in which all rows for a given type are combined. I.e. an output that looks like

array([[1., 0., 0., 0., 1., 1., 1., 1.],
       [0., 1., 0., 0., 1., 1., 1., 0.],
       [0., 0., 1., 0., 1., 1., 0., 0.],
       [0., 0., 0., 1., 1., 0., 0., 0.]])

where all feats for a given type are switched on, but there is only one row per feat. Is ColumnTransformer the right way to do that? Can I do it with only OneHotEncoder?

I don't care about the identity matrix on the left. Really what I'm after is just

array([[1., 1., 1., 1.],
       [1., 1., 1., 0.],
       [1., 1., 0., 0.],
       [1., 0., 0., 0.]])

I've tried playing around with ColumnTransformer or using just OneHotEncoder with something like

ohe = OneHotEncoder(categories=X['feat'].unique())
for t in X['type'].unique():
    print(t)
    print(ohe.fit_transform(X[X['type'] == t]).toarray())

with no success.

user_14277722 · Answer

If you are already working in pandas you can use get_dummies, groupby 'type' and aggregate with max for multi label encoding.

pd.get_dummies(X, columns=['feat']).groupby('type', as_index=False).max()

Output

    type  feat_A  feat_B  feat_C  feat_D
0  type1       1       1       1       1
1  type2       1       1       1       0
2  type3       1       1       0       0
3  type4       1       0       0       0

With sklearn you can use MultiLabelBinarizer. But you have to prepare your data with pandas anyway and gather the labels in a list. In my version of sklearn MultiLabelBinarizer does not work with ColumnTransformer because of a mismatch in the signature of the transform methods.

from sklearn.preprocessing import MultiLabelBinarizer

MultiLabelBinarizer().fit_transform(X.groupby('type')['feat'].apply(list))

Output

array([[1, 1, 1, 1],
       [1, 1, 1, 0],
       [1, 1, 0, 0],
       [1, 0, 0, 0]])

Alternatively, you can first use OneHotEncoder and then group and aggregate in pandas.

column_trans = ColumnTransformer(
    [('feate_', OneHotEncoder(),['feat'])],
    remainder='passthrough')

pd.DataFrame(column_trans.fit_transform(X)).groupby(4).max().set_axis([list('ABCD')], 1)

Output

         A    B    C    D
4                        
type1  1.0  1.0  1.0  1.0
type2  1.0  1.0  1.0  0.0
type3  1.0  1.0  0.0  0.0
type4  1.0  0.0  0.0  0.0

Combine rows in scikit during ColumnTransformer transform

1 Answers