Issues with hyperparameter tuning accuracy

user_18744526 0
Viewed 42

I'm currently trying to tune the hyperparameters of a decision tree model but I can't get a test set accuracy higher than the original (untuned model)

This is the code for the untuned model:

from sklearn.tree import DecisionTreeClassifier
modeldt = DecisionTreeClassifier(random_state=42)
modeldt.fit(input_train, target_train)
modeldt.score(input_train, target_train), modeldt.score(input_test, target_test)

and the output for this is (0.998062015503876, 0.7384615384615385)

and after adding a few tuned hyperparameters like so:

modeldt = DecisionTreeClassifier(max_depth=8, min_samples_leaf = 6, max_leaf_nodes = 11,splitter= 'best', criterion='gini', random_state=42)
modeldt.fit(input_train, target_train)
modeldt.score(input_train, target_train), modeldt.score(input_test, target_test)

the output for this is (0.7558139534883721, 0.7384615384615385)

as you can see only the training set accuracy changed but the test set stayed exactly same. I'm not sure what is going on or if there's something wrong with the input data...

hyperparameters machine-learning python
1 Answers

From the scores we can see that the untuned model is overfitted. Parameters such as max_depth, min_samples_leaf, max_leaf_nodes restrict the model. That is why the accuracy for the training subset decreased.

Regarding the score for the test subset - score function calculate the accuracy, and the accuracy can be tricky if you dataset is not balanced. In this case, you need to investigate your test set. As a debugging option, you can try to predict a few single examples and evaluate the quality of your model.

user_86093300
Related