how to change Beta value when using "classif.fbeta" as a performance measure in mlr3?

user_15519965 0
Viewed 142

library(mlr3verse)
preformace_msr <- msr("classif.fbeta", beta = 1.5)

I am trying to use custom value of BETA in fbeta measure for classification model tuning.

But the above way of trying to give beta value throws an error in mlr3.

What is the right way of doing it in mlr3?

mlr3
2 Answers

So the error was as follows:

library(mlr3verse)
preformace_msr <- msr("classif.fbeta", beta = 1.5)
#> Error: Cannot set argument 'beta' for 'MeasureBinaryimple' (not a constructor argument, not a parameter, not a field.

I think this has to do with the fact that only the F1-Measure (not F1.5-measure etc.) has been implemented in mlr3. See beta=1 in the source file.

In the mlr3 book you will find following under 6.3:

In this section we showcase how to implement a custom performance measure.

According to this section, you could implement an Fx-Measure with (I think my syntax is not the best/safest, but it should work):

library(mlr3verse)
library(mlr3measures)
library(R6)

# make custom measure:
MeasureCustomFbeta = R6::R6Class("classif.custom_fbeta",
                             inherit = mlr3::MeasureClassif,
                             public = list(
                               #declase field
                               beta=NULL, #delcare field
                               initialize = function(beta) {
                                 self$beta <- beta
                                 super$initialize(
                                   # custom id for the measure
                                   id = "custom_fbeta",
                                  
                                   # required predict type of the learner
                                   predict_type = "response",
                                   
                                   # feasible range of values
                                   range = c(0, Inf),
                                   
                                   # minimize during tuning?
                                   minimize = TRUE
                                 )
                               }
                             ),
  
                             
                             private = list(
                               # customized scoring function operating on the prediction object
                               .score = function(prediction, ...) {
                                 fbeta_cm = function(m, beta) {
                                   pred_pos = sum(m[1L, ])
                                   cond_pos = sum(m[, 1L])
                                   if (m[1L, 1L] == 0L || pred_pos == 0L || cond_pos == 0L)
                                     return(na_value)
                                   
                                   P = m[1L, 1L] / pred_pos
                                   R = m[1L, 1L] / cond_pos
                                   ((1 + beta^2) * P * R) / ((beta^2 * P) + R)
                                 }
                                 fbeta_cm(confusion_matrix(prediction$truth, prediction$response, prediction$positive)$matrix, self$beta)
                                 
                               })
                               
)

# add it to the dictionary
mlr3::mlr_measures$add("classif.custom_fbeta", MeasureCustomFbeta)

Test:

# get data
data("Sonar", package = "mlbench")

# make task
task = TaskClassif$new(id = "Sonar", Sonar, target = "Class", positive = "R")

# make learner
learner = lrn("classif.rpart", predict_type = "response")

# predict
pred = learner$train(task)$predict(task)
pred$confusion
#>         truth
#> response  R  M
#>        R 87 16
#>        M 10 95

# measure "classif.beta
measure_old <- msr("classif.fbeta")
pred$score(measure_old)
#> classif.fbeta 
#>          0.87

# customized measure
measure_new <- msr("classif.custom_fbeta", beta=1.5)
pred$score(measure_new)
#> classif.custom_fbeta 
#>            0.8801556
user_114821100
user_15204000
Related