I have been experimenting with python and columnTransformer and I have a question.
This is the code that works. It does what I want, fills the empty cells with most frequent and uses onehotencoding, so the columns (only binary features I have) get doubled after OneHotEncoding, as it should. Every cell value is in string.
cate_transformer = Pipeline(steps=[
('imputer3',SimpleImputer(missing_values='', strategy='most_frequent')),
('onehot3',OneHotEncoder(sparse=False))
])
ColumnTransformer(transformers=[
('num', cate_transformer, binary_cols)
])
X = prepro.fit_transform(X)
This also works, it gives me the exact same result, it's without the ColumnTransformer:
BinImputer = SimpleImputer(missing_values='', strategy='most_frequent')
X = BinImputer.fit_transform(X)
Hot = OneHotEncoder(sparse=False)
X = Hot.fit_transform(X)
Now, to my questions... This does not work:
column_trans = ColumnTransformer(
remainder='passthrough',
transformers=[
('SimpleImputer', SimpleImputer(missing_values='', strategy='most_frequent'), binary_cols)
])
column_trans1 = ColumnTransformer(
remainder='passthrough',
transformers=[
('OneHotEncoder', OneHotEncoder(sparse=False), binary_cols)
])`
X = column_trans.fit_transform(X)
X = column_trans1.fit_transform(X)
It says: ValueError: not enough values to unpack (expected 3, got 2)
Also, this does not work: When I pass a dataframe with x rows and 2 columns, it returns x rows and 8 columns. Look:
column_trans1 = ColumnTransformer(
remainder='passthrough',
transformers=[
('SimpleImputer', SimpleImputer(missing_values='', strategy='most_frequent'), binary_cols),
('OneHotEncoder', OneHotEncoder(sparse=False), binary_cols)
])
X = column_trans1.fit_transform(X)
Before: enter image description here
After: enter image description here
1)It seems very strange to me. Why at the last code, the total number of columns is x4 and not x2, as it should in Onehotencoding. Also: 2)Why at the last code, the code runs (even if it gives me these 8 columns, instead of 4), whereas at the previous, where I do the same thing but in 2 steps (or I feel I do the same thing), it does not run at all?
So, I must be missing something important on the way they are implemented. I tried looking at the documentations, but I could not get it. Thank you in advance.