Why is my predictive model accuracy is too high?

user_13244401 0
Viewed 76

This is my code for the train_test_split 

from sklearn.model_selection import train_test_split
X_train,X_test, y_train, y_test = train_test_split(X,y, test_size= 0.20, random_state = 40)
print("x_train ",X_train.shape)
print("x_test ",X_test.shape)
print("y_train ",y_train.shape)
print("y_test ",y_test.shape)

x_train  (32408, 29)
x_test  (8103, 29)
y_train  (32408,)
y_test  (8103,)

from sklearn.ensemble import RandomForestClassifier
classifier = RandomForestClassifier(random_state=0, n_estimators=100,\
                                   criterion = 'entropy', max_leaf_nodes=30,n_jobs=-1)
model_RF = classifier.fit(X_train, y_train)

acc_train_rf = round(classifier.score(X_train, y_train),2)*100
print(" Model accuracy within training data is : " +  str(acc_train_rf) +"%")

Model accuracy within training data is: 100.0%
machine-learning python
2 Answers

You are using your training data to calculate the score. Use your test data.

Change 

acc_train_rf = round(classifier.score(X_train, y_train),2)*100

to 

acc_train_rf = round(classifier.score(X_test, y_test),2)*100
user_131768960

I encountered the same problem before. My model finished the training with default params extremely fast and was always giving 100% on everything(accuracy/recall/precision/F1 score) no matter on train, valid, or test set.

My advice(s):

  • [priority] check if data leakage exists in your dataset.
  • there may be a very strong feature that can directly predict the target, do more EDA to better know your data.
  • check the structure of your dataset, the way of constructing your dataset may be problematic.
  • look at other metrics like recall/precision/F1 etc to see if all of them are 1.00.
  • model may be overfitting, check label distribution, i.e., do a y.value_counts() to explore your target
  • generate feature importance to look at if you have too many useless features.
user_78933870
Related