ValueError: Found unknown categories ['?'] in column 7 during transform - Breast Cancer Dataset

Viewed 3548

I am trying to work my way through a classification machine learning problem using the UCI reposity for rates of Breast cancer 1988 (https://archive.ics.uci.edu/ml/datasets/Breast+Cancer). I keep getting the following error, although not consistently. Sometimes the algorithm runs straight through to training the model and predicting test accuracy, sometimes it fails on OneHotEncoding and shows the following error:

ohe = OneHotEncoder()
ohe.fit(X_train)
X_train_encoded = ohe.transform(X_train)
X_test_encoded = ohe.transform(X_test)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-5-2cfd638a5b4d> in <module>()
      2 ohe.fit(X_train)
      3 X_train_encoded = ohe.transform(X_train)
----> 4 X_test_encoded = ohe.transform(X_test)

1 frames
/usr/local/lib/python3.6/dist-packages/sklearn/preprocessing/_encoders.py in _transform(self, X, handle_unknown)
    122                     msg = ("Found unknown categories {0} in column {1}"
    123                            " during transform".format(diff, i))
--> 124                     raise ValueError(msg)
    125                 else:
    126                     # Set the problematic rows to an acceptable value and

ValueError: Found unknown categories ['?'] in column 7 during transform

I have tried running in both Colab and Spyder and get the same issue, not sure where I am going wrong. I am imputing missing values before splitting the dataset and then encoding, but even when I remove SimpleImputer I still get the error.

dataset = pd.read_csv('breast-cancer.csv')
X = dataset.iloc[:, :-1].values
y = dataset.iloc[:, -1].values

from sklearn.impute import SimpleImputer
imputer = SimpleImputer(missing_values=np.nan, strategy='most_frequent')
imputer.fit(X)
X_imputed = imputer.transform(X)

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X_imputed, y, test_size = 0.25)

ohe = OneHotEncoder()
ohe.fit(X_train)
X_train_encoded = ohe.transform(X_train)
X_test_encoded = ohe.transform(X_test)

<-- Code stops running here -->

le = LabelEncoder()
le.fit(y_train)
y_train_encoded = le.transform(y_train)
y_test_encoded = le.transform(y_test)
1 Answers

The test data might contain new entries not present in train data. Can you try this?

ohe = OneHotEncoder(handle_unknown = "ignore")

About this parameter : Whether to raise an error or ignore if an unknown categorical feature is present during transform (default is to raise). When this parameter is set to ‘ignore’ and an unknown category is encountered during transform, the resulting one-hot encoded columns for this feature will be all zeros.

more here :

https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html

Related