mlr3 predictions to new data with parameters from autotune

user_6357433 0
Viewed 654

I have a follow-up question to this one. As in the initial question, I am using the mlr3verse, have a new dataset, and would like to make predictions using parameters that performed well during autotuning. The answer to that question says to use at$train(task). This seems to initiate tuning again. Does it take advantage of the nested resampling at all by using those parameters?

Also, looking at at$tuning_result there are two sets of parameters, one called tune_x and one called params. What is the difference between these?

Thanks.

Edit: example workflow added below

library(mlr3verse)

set.seed(56624)
task = tsk("mtcars")

learner = lrn("regr.xgboost")

tune_ps = ParamSet$new(list(
  ParamDbl$new("eta", lower = .1, upper = .4),
  ParamInt$new("max_depth", lower = 2, upper = 4)
))

at = AutoTuner$new(learner = learner, 
                   resampling = rsmp("holdout"), # inner resampling
                   measures = msr("regr.mse"), 
                   tune_ps = tune_ps,
                   terminator = term("evals", n_evals = 3),
                   tuner = tnr("random_search"))

rr = resample(task = task, learner = at, resampling = rsmp("cv", folds = 2),
              store_models = TRUE)
rr$aggregate()
rr$score()
lapply(rr$learners, function(x) x$tuning_result)

at$train(task)
at$tuning_result

notreallynew.df = as.data.table(task)
at$predict_newdata(newdata = notreallynew.df)
mlr3 r
1 Answers

As ?AutoTuner tells, this class fits a model with the best hyperparameters found during the tuning. This model is then used for prediction, in your case to newdata when calling its method .$predict_newdata().

Also in ?AutoTuner you see the documentation linked to ?TuningInstance. This then tells you what the $tune_x and params slots represent. Try to look up the help pages next time - that's what they are there for ;)

This seems to initiate tuning again.

Why again? It does it in the first place, on all observations of task. I assume you might confuse yourself by the common misconception between "train/predict" vs. "resample". Read more about the theoretical differences of both to understand what both are doing. They have completely different aims and are not connected. Maybe the following reprex makes it more clear.

library(mlr3verse)
#> Loading required package: mlr3
#> Loading required package: mlr3db
#> Loading required package: mlr3filters
#> Loading required package: mlr3learners
#> Loading required package: mlr3pipelines
#> Loading required package: mlr3tuning
#> Loading required package: mlr3viz
#> Loading required package: paradox

set.seed(56624)
lgr::get_logger("mlr3")$set_threshold("warn")
task = tsk("mtcars")
learner = lrn("regr.xgboost")

tune_ps = ParamSet$new(list(
  ParamDbl$new("eta", lower = .1, upper = .4),
  ParamInt$new("max_depth", lower = 2, upper = 4)
))

at = AutoTuner$new(
  learner = learner,
  resampling = rsmp("holdout"), # inner resampling
  measures = msr("regr.mse"),
  tune_ps = tune_ps,
  terminator = term("evals", n_evals = 3),
  tuner = tnr("random_search"))

# train/predict with AutoTuner
at$train(task)
notreallynew.df = as.data.table(task)
at$predict_newdata(newdata = notreallynew.df)
#> <PredictionRegr> for 32 observations:
#>     row_id truth response
#>          1  21.0 9.272631
#>          2  21.0 9.272631
#>          3  22.8 9.272631
#> ---                      
#>         30  19.7 9.272631
#>         31  15.0 5.875841
#>         32  21.4 9.272631

# resample with AutoTuner for performance estimation

rr = resample(
  task = task, learner = at, resampling = rsmp("cv", folds = 2),
  store_models = TRUE)
rr$aggregate()
#> regr.mse 
#> 240.5866
rr$score()
#>          task task_id     learner         learner_id     resampling
#> 1: <TaskRegr>  mtcars <AutoTuner> regr.xgboost.tuned <ResamplingCV>
#> 2: <TaskRegr>  mtcars <AutoTuner> regr.xgboost.tuned <ResamplingCV>
#>    resampling_id iteration prediction regr.mse
#> 1:            cv         1     <list> 220.8076
#> 2:            cv         2     <list> 260.3656

Created on 2020-05-07 by the reprex package (v0.3.0)

user_41857850
Related