Ordinal encoder issues with NaN values

user_1960266 0
Viewed 4836

I have a dataframe with blank spaces as missing values, so I have replaced them with NaN values by using a regex. The problem that I have is when I want to use ordinal encoding for replacing categorical values. My code so far is the following:

    x=pd.DataFrame(np.array([30,"lawyer","France",
                             25,"clerk","Italy",
                             22," ","Germany",
                             40,"salesman","EEUU",
                             34,"lawyer"," ",
                             50,"salesman","France"]
                             
            ).reshape(6,3))
    x.columns=["age","job","country"]
    x = x.replace(r'^\s*$', np.nan, regex=True)

    oe=preprocessing.OrdinalEncoder()
    df.job=oe.fit_transform(df["job"].values.reshape(-1,1))

I got the following error:

Input contains NaN

I would like that the job column gets replaced with numbers such as: [1,2,-1,3,1,3].

pandas ordinal python
3 Answers

You can try with factorize, notice here is category start with 0

x.job.mask(x.job==' ').factorize()[0]
Out[210]: array([ 0,  1, -1,  2,  0,  2], dtype=int32)
user_79645270

sklearn.preprocessing.OrdinalEncoder doesn't allow NaN. If you want to use it, you need to drop NaN before fetching to OrdinalEncoder, assign the result back to the column and fillna

from sklearn import preprocessing

oe = preprocessing.OrdinalEncoder()
x.loc[x.job.notna(), ['job']] = oe.fit_transform(x["job"].dropna().values.reshape(-1,1))
x['job'] = x.job.fillna(-1)

Out[52]:
  age  job  country
0  30  1.0   France
1  25  0.0    Italy
2  22 -1.0  Germany
3  40  2.0     EEUU
4  34  1.0      NaN
5  50  2.0   France

Note: clerk is ordinal ordering before lawyer, so its ordinal-encoded value must smaller lawyer. So clerk is 0 and lawyer is 1

user_101892140

Here is a solution if you want to use None values as the 0th index of the category

Note: Here I have the same list of category items so I used the same list for each column, you might have different lists

all_categories = data[["columns", "you", "want"]].values.ravel()
all_categories = [i for i in all_categories if pd.notnull(i)]
unique_categories = [None] + list(pd.unique(all_categories))

oe_cat = OrdinalEncoder(handle_unknown='use_encoded_value', unknown_value=-1, categories=[unique_categories for _ in ["columns", "you", "want"]])
oe_cat.fit(merged[["columns", "you", "want"]])
user_58843800
Related