Label encoding with possible unseen data

user_5740200 0
Viewed 1289

I have a dataframe with multiple columns which need to be label-encoded. Problem is that the test group may include unseen data (classes) in the future. I'd like those classes to be labeled as a group of their own so that the code won't crash when I predict new data sets.

I tried using sklearn labelencoder but received.

ValueError: y contains previously unseen labels: 'rat'

I also need the encoder to be reusable, meaning that I'll be able to encode future datasets with the same values.

Is there a way to do that?

xgboost machine-learning python
3 Answers

I have faced same difficulty a multiple times.

My workaround is a bit expensive though

le=LabelEncoder()
le.fit(trainDf)

le.classes_=np.array([-99999] + le.classes_.tolist())
testDf[~testDf.isin(le.classes_)]=-99999 #anything that is not used in your dataframe and the same datatype (here int64)

le.transform(testDf)
user_137178510

I slightly updated Sayan Dey's idea:

Step 1: label encoding the calsses which exist in the label encoder. Step 2: fitting the label encoder then setting to -1 all classes in test which are NOT in the encoder.

i='browser'
le = LabelEncoder()
train[i] = le.fit_transform(train[i])
#Set classes in test which don't exist in the encoder to -1
test.loc[~test[i].isin(le.classes_),i] = -1    
#Encode classes that exist in the encoder
test.loc[test[i].isin(le.classes_),i] = le.transform(test[i][test[i].isin(le.classes_)])
user_57402000

Can be solved with scikit-learn is using the class OneHotEncoder with parameter handle_unknown='ignore':

from sklearn.preprocessing import OneHotEncoder

encoder = OneHotEncoder(handle_unknown='ignore')
encoder.fit(train)

encoder.transform(train)
user_86093300
Related