Scikit-learn's (0.22.1) KNNImputer returning wrong number of values

user_10101235 0
Viewed 796

My original data set shape is (790215,20) which contains features that have around 60-80% missing values. I decided to use scikit-learn's KNNImputer as follows

import pandas as pd
from sklearn.impute import KNNImputer

# Loading the data
dataset = pd.read_csv('Dataset.csv')

# To avoid 'MemoryError' imputing individually

#Loading the first feature
X = dataset.iloc[:,0].values

#Imputing with default parameters 
imputer = KNNImputer()

#Reshaping to meet the dimensional requirement
X_imp  = imputer.fit_transform(X.reshape(1,-1))

Now the shape of X_imp is (1,729026)

I'm not sure what I did wrong. Why the 790215 changed to 729026.

Update :

The X.shape is (790215,)

X.reshape(1,-1).shape is (1,790215)

X.reshape(1,-1) is array([[ nan, 97., 89., ..., 140., 120., 115.]])

scikit-learn pandas numpy python
1 Answers

The way you used reshape is the problem. You had converted your data into single datapoint by giving .reshape(1, -1). Meaning 1 row with 790215 columns. Hence while transforming the KNNImputer drops the columns which has only nan values. This is the reason for the drop.

Instead you need to use .reshape(-1,1), which will make it as 790215 rows and 1 column.

Note: Using single feature for KNNImputer might not work well. Better you can do it with 3-5 features at a time. Also you can take a look at SimpleImputer.

user_63476290
Related