Missing column labels after filling missing values in scikit-learn

user_17233625 0
Viewed 55

I was recently tried to fill missing data in a dataset after which the column labels in the dataset disappeared. The dataset is was using looks like:

Make Colour Odometer (KM) Doors
0 Honda White 35431.0 4.0
1 BMW Blue 192714.0 5.0
2 Honda White 84714.0 4.0
3 Toyota White 154365.0 4.0
4 Nissan Blue 181577.0 3.0

The method I used is

from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer

cat_imputer=SimpleImputer(strategy='constant',fill_value='missing')
door_imputer=SimpleImputer(strategy='constant',fill_value=4)
num_imputer=SimpleImputer(strategy="mean")

cat_features=["Make","Colour"]
door_features=["Doors"]
num_features=["Odometer (KM)"]

imputer = ColumnTransformer([
    ("cat_imputer",cat_imputer,cat_features),
    ("door_imputer",door_imputer,door_features),
    ("num_imputer",num_imputer,num_features)
])

filled_x=imputer.fit_transform(X)
new_frame=pd.DataFrame(filled_x)

After this, if we view the new_frame it is

0 1 2 3
0 Honda White 4.0 35431.04.0
1 BMW Blue 5.0 192714.0
2 Honda White 4.0 84714.0
3 Toyota White 4.0 154365.0
4 Nissan Blue 3.0 181577.0

So what happened here?

Why are the Doors and Odometer (KM) columns flipped?
Where did the column labels go?
How can I bring them back?

imputation scikit-learn pandas dataframe python
1 Answers

because it generates a whole new filled_x array, which the type is numpy.ndarray, either you assign them back to the original dataframe or to a new dataframe equipt with old dataframe's columns and index.

df
###
     Make Colour  Odometer (KM)  Doors
0   Honda  White        35431.0    4.0
1     BMW   Blue       192714.0    5.0
2   Honda    NaN        84714.0    4.0
3  Toyota  White       154365.0    4.0
4  Nissan   Blue       181577.0    NaN

I remodify your df with some NaN, so we can check if the imputer works or not.




Imputation

Why are the Doors and Odometer (KM) columns flipped?

Your imputer's orders differ from your original dataframe's column order.

cat_imputer=SimpleImputer(strategy='constant',fill_value='missing')
door_imputer=SimpleImputer(strategy='constant',fill_value=4)
num_imputer=SimpleImputer(strategy="mean")

cat_features=["Make","Colour"]
door_features=["Doors"]
num_features=["Odometer (KM)"]

imputer = ColumnTransformer([
    ("cat_imputer",cat_imputer,cat_features),
    ("num_imputer",num_imputer,num_features),     ◀━┓
    ("door_imputer",door_imputer,door_features)   ◀━┛
])

filled_x=imputer.fit_transform(df)

filled_x
###
[['Honda' 'White' 35431.0 4.0]
 ['BMW' 'Blue' 192714.0 5.0]
 ['Honda' 'missing' 84714.0 4.0]
 ['Toyota' 'White' 154365.0 4.0]
 ['Nissan' 'Blue' 181577.0 4.0]]

During the processing, sklearn are dealing with array/ndarray

type(filled_x)
###
numpy.ndarray

you use pd.DataFrame(filled_x) that made the ndarray-format result from imputer.fit_transform() putting into a brand new dataframe without specifying columns and index information.

You may assign them like:

new_frame = pd.DataFrame(filled_x, index=df.index, columns=df.columns)
new_frame
###

enter image description here

And this is a way(though it's not recommended) to assign the result to overwrite the old df

df.loc[:,:] = filled_x

Practically speaking, we might reserve df so you can try different imputation methods.

user_145278860
Related