Imputer on some columns in a Dataframe

user_10379765 0
Viewed 6850

I am trying to use Imputer on a singe column called age to replace missing values.But I get the error as " Expected 2D array, got 1D array instead:"

Following is my code

import pandas as pd
import numpy as np
from sklearn.preprocessing import Imputer

dataset = pd.read_csv("titanic_train.csv")

dataset.drop('Cabin',axis = 1,inplace = True)
x = dataset.drop('Survived',axis = 1)
y = dataset['Survived']

imputer = Imputer(missing_values ="nan",strategy = "mean",axis = 1)
imputer=imputer.fit(x['Age'])
x['Age']=imputer.transform(x['Age'])
imputation scikit-learn pandas machine-learning python
3 Answers

The Imputer is expecting a 2-dimensional array as input, even if one of those dimensions is of length 1. This can be achieved using np.reshape:

imputer = Imputer(missing_values='NaN', strategy='mean')
imputer.fit(x['Age'].values.reshape(-1, 1))
x['Age'] = imputer.transform(x['Age'].values.reshape(-1, 1))

That said, if you are not doing anything more complicated than filling in missing values with the mean, you might find it easier to skip the Imputer altogether and just use Pandas fillna instead:

x['Age'].fillna(x['Age'].mean(), inplace=True)
user_68668110

Although @thesilkworkm beat me in the curb, it may be useful to know why exactly your own code doesn't work.

So, apart from the reshape issue, there are two more mistakes in your code; the first is that you erroneously ask for axis=1 in your imputer, while you should ask for axis=0 (which is the default value, and that's why it works when omitted completely, as in @thesilkworkm'a answer); from the docs:

axis : integer, optional (default=0)

The axis along which to impute.

  • If axis=0, then impute along columns.
  • If axis=1, then impute along rows.

The second mistake is your missing_values argument, which should be 'NaN', and not 'nan'; from the docs again:

missing_values : integer or “NaN”, optional (default=”NaN”)

The placeholder for the missing values. All occurrences of missing_values will be imputed. For missing values encoded as np.nan, use the string value “NaN”.

So, just for offering an alternative but equivalent solution (beyond the one already provided by @thesilkworm), you can also fit & transform in one line:

imp = Imputer(missing_values ="NaN",strategy = "mean",axis = 0)
x['Age'] = imp.fit_transform(x['Age'].reshape(-1,1))
user_46854710

When you are fit tranforming it use reshape(-1,1). Because method is expecting a 2D array as input but you are giving 1D array.

Ex: x['Age']=imputer.transform(x['Age'].reshape(-1,1))

user_102600420
Related