I am using grid search to optimize the hyper-parameters of a Random Forest fit on a balanced data set, and I am struggling with which model evaluation metric to choose. Given the real-world context of this problem, false negatives are more costly than false positives. I initially tried optimizing recall but I was ending up with extremely high numbers of false positives. My solution is to instead optimize an f-beta score with beta > 1. My question is, how best to choose beta? If I can calculate the cost of a false negative and false positive, can I set beta = Cost of False Negative/Cost of False Positive? Does this approach make sense?
1 Answers
- To give more weight to the Precision, we pick a Beta value in the interval 0 < Beta < 1
- To give more weight to the Recall, we pick a Beta Value in the interval 1 < Beta
When you set beta = Cost of False Negative/Cost of False Positive then you'll give more weight to recall, in case of the cost of False negative is higher than that of False positive, so it will work, but this doesn't mean that this is the optimum solution for your problem.
Optimizing Beta is relevant to the shape of your data, so it would be better to try different values of Beta on your data until you get the best value.
Related