Which model to pick from K fold Cross Validation

user_4966945 0
Viewed 11803

I was reading about cross validation and about how it it is used to select the best model and estimate parameters , I did not really understand the meaning of it.

Suppose I build a Linear regression model and go for a 10 fold cross validation, I think each of the 10 will have different coefficiant values , now from 10 different which should I pick as my final model or estimate parameters.

Or do we use Cross Validation only for the purpose of finding an average error(average of 10 models in our case) and comparing against another model ?

statistics-bootstrap cross-validation machine-learning validation statistics
5 Answers

If your build a Linear regression model and go for a 10 fold cross validation, indeed each of the 10 will have different coefficient values. The reason why you use cross validation is that you get a robust idea of the error of your linear model - rather than just evaluating it on one train/test split only, which could be unfortunate or too lucky. CV is more robust as no ten splits can be all ten lucky or all ten unfortunate.

Your final model is then trained on the whole training set - this is where your final coefficients come from.

user_64986380

Cross Validation is mainly used for the comparison of different models. For each model, you may get the average generalization error on the k validation sets. Then you will be able to choose the model with the lowest average generation error as your optimal model.

user_106608800

Cross-Validation or CV allows us to compare different machine learning methods and get a sense of how well they will work in practice.

Scenario-1 (Directly related to the question)

  • Yes, CV can be used to know which method (SVM, Random Forest, etc) will perform best and we can pick that method to work further.

(From these methods different models will be generated and evaluated for each method and an average metric is calculated for each method and the best average metric will help in selecting the method)

  • After getting the information about the best method/ or best parameters we can train/retrain our model on the training dataset.
  • For parameters or coefficients, these can be determined by grid search techniques. See grid search

Scenario-2:

Suppose you have a small amount of data and you want to perform training, validation and testing on data. Then dividing such a small amount of data into three sets reduce the training samples drastically and the result will depend on the choice of pairs of training and validation sets. CV will come to the rescue here. In this case, we don't need the validation set but we still need to hold the test data. A model will be trained on k-1 folds of training data and the remaining 1 fold will be used for validating the data. A mean and standard deviation metric will be generated to see how well the model will perform in practice.

user_96127020
Related