Difference between K fold and Iterated K fold with shuffling

user_8619880 0
Viewed 1877

K-FOLD VALIDATION:

In this you split your data into K partitions of equal size. For each partition i, train a model on the remaining K – 1 partitions, and evaluate it on partition i.

Your final score is then the averages of the K scores obtained. This method is helpful when the performance of your model shows significant variance based on your train test split.

ITERATED K-FOLD VALIDATION WITH SHUFFLING:

This is for situations in which you have relatively little data available and you need to evaluate your model as precisely as possible.

It consists of applying K-fold validation multiple times, shuffling the data every time before splitting it K ways. The final score is the average of the scores obtained at each run of K-fold validation

I could not understand an implement iterated k fold if any one can help with an example code or flow chart showing differences between k-fold and iterated K-fold.

deep-learning python
2 Answers

It's worth mentioning that the first 3 paragraphs of the question body are a summary of text from chapter 4.2.1 of Deep Learning with Python by Francois Chollet. Unfortunately, the author doesn't explain the concept of Iterated K-fold validation with shuffling in greater detail.

I found this explanation which hopefully makes the concept clearer:

3-fold cross validation steps:

  1. shuffle data
  2. split your data into k = 3 segments (aka folds)
  3. for each fold: train a model without the fold in question and test on the left-out fold
  4. after all k = 3 folds are done, calculate the average test accuracy (= of all 3 folds)

Iterated K-fold cross validation (aka repeated k-fold cross validation) repeats/iterates the process described in steps 1-4 a chosen number of times (e.g. 100 times). You now have 100 average scores (each as a result of repeatedly applying steps 1-4). You take the average of those 100 scores and this final result will represent your final test accuracy obtained in a robust way, your model being evaluated 'as precisely as possible'.

They key difference between K-fold validation and Iterated K-fold validation with shuffling seems to be that in the latter steps 1-4 are performed repeatedly and then an average is taken (representing the final test accuracy), whereas in the former steps 1-4 are performed once (not necessarily including step 1) and the score obtained at step 4 represents your final test accuracy.

The explanation was closely adapted from here with thanks to user @cbeleites supports Monica.

user_26161850

I also arrived here after reading Deep Learning with Python by Francois Chollet. I implemented iterated K-fold cross-validation in my own code and am posting here in case it helps others.

The difference between the two is whether or not you implement an outer loop to run K-fold cross-validation many times. The intention is to compute an average of averages that may be a more accurate representation of how your model will perform on the test set. Hope this helps. 

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import StratifiedKFold

model = RandomForestClassifier()

N_iterations = 10
N_folds = 5

# Initialize array to store the scores of each K-fold cross-validation
scores = np.zeros((N_iterations, N_folds))

# This loop is what makes it iterated K-fold validation
for i in range(N_iterations):

    skf = StratifiedKFold(n_splits=N_folds, shuffle=True)

    # This loops is traditional K-fold cross validation
    for j, (train_index, test_index) in enumerate(skf.split(X_train, y_train), 0):

        X_tr = X_train[train_index]
        X_te = X_train[test_index]

        y_tr = y_train[train_index]
        y_te = y_train[test_index]

        model.fit(X_tr, y_tr)

        scores[i][j] = model.score(X_te, y_te)

    print("Iteration", i+1, "avg: %.4f" % np.mean(scores[i]))

print("\nAvg score: %.4f" % np.mean(np.mean(scores, axis=1)))
user_41670920
Related