Using Caret to select features within folds of a cross validation

Viewed 751

In the caret package, is there any way to use the recursive feature elimination function within the folds of a cross validation scheme in a trainControl that passes to a train function which uses tuning grids?

I love the recursive feature elimination function, but it really should be applied to the training folds in cross validation and then tested on the hold-out fold.

I've played around with a bunch of different methods to do this, but none are perfect. For example, I can make my own cross validation folds and run trainControl with method = 'none' but that won't utilize the training grid in train (an evaluation group is needed for that). I can also make my own cv folds, and have method = 'cv', in the trainControl (I can use a tuning grid here), but the best tune is chosen on the hold-out sample generated by the trainControl hold-out, not my hold-out.

Is there a way to tell caret to evaluate the models with the tuning grids on my pre-specified hold-out fold (the one taken prior to feature elimination)?

In my work-flow, I am testing a few different model types with their own tuning grids. There are parts of caret I really like, and I've spent a ton of time on this so I'd like to use it, but this is a deal breaker if I can't get it to work. I'm open to any suggestions!

Thanks in advance-

SOLUTION: My solution may not be the most efficient, but it seems to work. I made my cross validation folds using information here: https://stats.stackexchange.com/questions/61090/how-to-split-a-data-set-to-do-10-fold-cross-validation. Using the createFolds (caret function) does not create equal folds, so I opted for the second solution. It looks like you may could do it with caret's stratified sampling, but I haven't explored that yet.

This code utilizes a bootstrap approach within each cv fold and predicts all of the observations in the hold out fold for each iteration.

  ## Make the folds for the cross validation
  folds <- cut(seq(1,nrow(data)), breaks=10, labels=FALSE) %>%
    sample(., length(.), replace= F)

  for(f in 1:10) { 

    testIndexes <- which(folds == f,arr.ind=TRUE)
    trainIndexes <- which(folds != f, arr.ind= T)

    trainIndexList <- replicate(500, sample(trainIndexes, length(trainIndexes), replace = T), simplify = F)
    testIndexList <- replicate(500, testIndexes, simplify = F)

    testData <- data[testIndexes, ]
    trainData <- data[-testIndexes, ]

    ## Make the train control object
    train_control <- trainControl(method = 'boot', 
                                  numbe r= 1,
                                  summaryFunction = modfun,
                                  preProcOptions = c('center', 'scale', newdata= testData),
                                  index = trainIndexList,
                                  indexOut = testIndexList,
                                  classProbs = T,
                                  savePredictions = T)

  ## Feature Selection
    ## Make the control for the recursive feature elimination
  rfe_control <- rfeControl(functions = rfFuncs, method = 'cv', number= 10)

    ## Generate the data frame based on feature selection
  fs_results <- rfe(trainData[,2:ncol(trainData)],
                    trainData[,'target'],
                    sizes=c(2:ncol(trainData)),
                    rfeControl= rfe_control)

  use_features <- c('target', predictors(fs_results))

  features <- predictors(fs_results) %>% data.frame(features= .) %>% mutate(fold= f) %>%
    rbind(features, .) ## Specify features as a data frame ahead of time

  data_min <- data[, use_features] %>% data.frame()

...(modeling code, including train functions and desired output)...

}

I haven't tried to do an lapply instead of a for loop yet. I'd appreciate any suggestions for efficiency.

0 Answers
Related