I am trying to work my way through a classification machine learning problem using the UCI reposity for rates of Breast cancer 1988 (https://archive.ics.uci.edu/ml/datasets/Breast+Cancer). I keep getting the following error, although not consistently. Sometimes the algorithm runs straight through to training the model and predicting test accuracy, sometimes it fails on OneHotEncoding and shows the following error:
ohe = OneHotEncoder()
ohe.fit(X_train)
X_train_encoded = ohe.transform(X_train)
X_test_encoded = ohe.transform(X_test)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-5-2cfd638a5b4d> in <module>()
2 ohe.fit(X_train)
3 X_train_encoded = ohe.transform(X_train)
----> 4 X_test_encoded = ohe.transform(X_test)
1 frames
/usr/local/lib/python3.6/dist-packages/sklearn/preprocessing/_encoders.py in _transform(self, X, handle_unknown)
122 msg = ("Found unknown categories {0} in column {1}"
123 " during transform".format(diff, i))
--> 124 raise ValueError(msg)
125 else:
126 # Set the problematic rows to an acceptable value and
ValueError: Found unknown categories ['?'] in column 7 during transform
I have tried running in both Colab and Spyder and get the same issue, not sure where I am going wrong. I am imputing missing values before splitting the dataset and then encoding, but even when I remove SimpleImputer I still get the error.
dataset = pd.read_csv('breast-cancer.csv')
X = dataset.iloc[:, :-1].values
y = dataset.iloc[:, -1].values
from sklearn.impute import SimpleImputer
imputer = SimpleImputer(missing_values=np.nan, strategy='most_frequent')
imputer.fit(X)
X_imputed = imputer.transform(X)
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X_imputed, y, test_size = 0.25)
ohe = OneHotEncoder()
ohe.fit(X_train)
X_train_encoded = ohe.transform(X_train)
X_test_encoded = ohe.transform(X_test)
<-- Code stops running here -->
le = LabelEncoder()
le.fit(y_train)
y_train_encoded = le.transform(y_train)
y_test_encoded = le.transform(y_test)