h2o cross-validated results can't be replicated

Viewed 39

I have been trying to replicate the results from cross-validation from h2o by hand, but I'm puzzled on why the results are different. In my understanding, if a fold_column is specified, then cross-validation happens according to the groups defined within that column. Therefore, for fold 1, the model should train on folds 2, 3, 4 and 5.

set.seed(1)
iris$fold <- sample(1:5, nrow(iris), replace = TRUE)

fit_cv <- h2o.glm(x = c("Species", "Petal.Width", "Petal.Length"),
                  y = c("Sepal.Length"),
                  training_frame = as.h2o(iris),
                  fold_column = "fold", 
                  alpha = 0.2, lambda = 0.1, seed=1,
                  keep_cross_validation_models = TRUE)

This outputs for fold 1 a training MAE of:

fit_fold1 <-  h2o.cross_validation_models(fit_cv)[[1]]

> fit_fold1@model$training_metrics@metrics$mae
[1] 0.3035526

However, if fitted by hand, the training MAE is (slightly) different:

dd_fold1_fit <- iris[iris$fold != 1,]

fit_fold1_direct <- h2o.glm(x = c("Species", "Petal.Width", "Petal.Length"),
                                y = c("Sepal.Length"),
                                training_frame = as.h2o(dd_fold1_fit),
                                alpha = 0.2, lambda = 0.1, 
                                seed=1)
    
fit_fold1_direct@model$training_metrics@metrics$mae
[1] 0.3004132

This is a minimal example, the issue is that with real data that I am working with the difference is larger, and I can't replicate the cross-validation results.

Any idea on what the explanation could be?

0 Answers
Related