Single class prediction error on certain folds of cross-validation in CNN

I am training a CNN to predict the correct orientation of x-ray. To further validate my model, i perform stratified K-fold validation. I use K=5 as a starting point. It turns out the training will eventually meet an epoch of 0.50 accuracy (because of single class prediction) once in a while.

Even though I can run the script till I get non of the problematic epoch, the problem still bugged me.

The 2 classes is splitted evenly between them in both training set and validation set. This is the confusion matrix of the classification.

Example of Bad Epoch
Example of Good Epoch

I need some enlightenment on how is this possible even though it is in the same script execution. What are the possibilities? I tried the model without cross validation, it works fine.

Convolutional Neural Network code

def get_model_cnn(input_shape=()):

    model = models.Sequential([
    layers.Conv2D(filters=32, kernel_size=(5, 5), strides=(1,  1), activation='relu', input_shape=input_shape),
    layers.MaxPooling2D(pool_size=(4, 4), strides=(4, 4), padding='valid'),

    layers.Conv2D(filters=32, kernel_size=(5, 5), strides=(1, 1), activation='relu'),
    layers.MaxPooling2D(pool_size=(3, 3), strides=(3, 3), padding='valid'),

    layers.Conv2D(filters=32, kernel_size=(3, 3), strides=(1, 1), activation='relu'),
    layers.MaxPooling2D(pool_size=(3, 3), strides=(3, 3), padding='valid'),

    layers.Flatten(),

    layers.Dense(128, activation='relu'),
    layers.Dense(2, activation='softmax')
])

    model.compile(optimizer=opt, loss='binary_crossentropy', metrics=['accuracy'])

    return model  

Stratified K-fold Code

skfold = StratifiedKFold(n_splits = 5, shuffle = True)

Model Training This code is simplified. I get rid of plotting codes.

for train_idx, val_idx in list(skfold.split(train_x,train_y)):

   x_train_df = df.iloc[train_idx]  
   x_valid_df   = df.iloc[val_idx]

   print(len(x_train_df))

   training_set = train_datagen.flow_from_dataframe(dataframe = x_train_df,
                                              x_col="image", 
                                              y_col="type",
                                              target_size= (IMAGE_H,IMAGE_L),
                                              batch_size = 64,
                                              color_mode= "grayscale", 
                                              class_mode= 'categorical',
                                              shuffle = True)


   validation_set = validation_datagen.flow_from_dataframe(dataframe = x_valid_df,
                                             x_col="image", 
                                             y_col="type",
                                             target_size= (IMAGE_H,IMAGE_L),
                                             batch_size = 64,
                                             color_mode= "grayscale",
                                             class_mode= 'categorical',
                                             shuffle = False)

   cnn = get_model_cnn(input_shape=(IMAGE_H, IMAGE_L, CHANNEL))

   history = cnn.fit(training_set, epochs=EPOCHS, validation_data = validation_set)
   backend.clear_session()