Like several people, I've been trying to use a sklearn.preprocessing.MultiLabelBinarizer within a sklearn.compose.ColumnTransformer pipeline, to little avail. Using some hints from answers in the linked thread, I came up with the following class, ComposableMultiLabelBinarizer. Everything runs without explicit error, but the results from the call are wrong as compared to a basic MultiLabelBinarizer's behavior:
import pandas as pd
from sklearn.preprocessing import MultiLabelBinarizer
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.compose import ColumnTransformer
class ComposableMultiLabelBinarizer(BaseEstimator, TransformerMixin):
def __init__(self):
self.mlb = MultiLabelBinarizer(sparse_output=False)
def fit(self, X, y=None):
self.parent_feature = X.columns[0]
self.mlb.fit(X)
return self
def transform(self, X):
return self.mlb.transform(X)
def get_feature_names(self):
return [f'{self.parent_feature}_{c}' for c in self.mlb.classes_]
df = pd.DataFrame({'x0': [['a', 'b', 'c'], ['c', 'b'], ['d']]})
transformer = ColumnTransformer([('x0', ComposableMultiLabelBinarizer(), ['x0'])])
transformer.fit_transform(df[['x0']])
# array([[1, 1]])
MultiLabelBinarizer().fit_transform(df['x0'])
# array([[1, 1, 1, 0],
# [0, 1, 1, 0],
# [0, 0, 0, 1]])
It is unclear to me why the single-transformer transformer doesn't appropriately handle this case, especially since I'm not passing any custom classes via MultiLabelBinarizer's classes option.