Accuracy reduced when shuffle set to True in Keras fit_generator

The data I am working with is very imbalanced.

I am training an image classifier using VGG16. I freezed all the layers in VGG16 accept the last two fully connected layers.

BATCH_SIZE = 128

EPOCHS = 80

When I set shuffle = False, the precision and recall for each class is very high (between .80-.90) but when I set shuffle = True, the precision and recall, for each class, drops to 0.10-0.20. I am not sure what is going on. Can some please help?

Below is the code:

img_size = 224
trainGen = trainAug.flow_from_directory(
    trainPath,
    class_mode="categorical",
    target_size=(img_size, img_size),
    color_mode="rgb",
    shuffle=False,
    batch_size=BATCH_SIZE)
valGen = valAug.flow_from_directory(
    valPath,
    class_mode="categorical",
    target_size=(img_size, img_size),
    color_mode="rgb",
    shuffle=False,
    batch_size=BATCH_SIZE)

testGen = valAug.flow_from_directory(
    testPath,
    class_mode="categorical",
    target_size=(img_size, img_size),
    color_mode="rgb",
    shuffle=False,
    batch_size=BATCH_SIZE)

baseModel = VGG16(weights="imagenet", include_top=False,input_tensor=Input(shape=(img_size, img_size, 3)))
headModel = baseModel.output
headModel = Flatten(name="flatten")(headModel)
headModel = Dense(512, activation="relu")(headModel)
headModel = Dropout(0.5)(headModel)
headModel = Dense(PFR_NUM_CLASS, activation="softmax")(headModel)
# place the head FC model on top of the base model (this will become
# the actual model we will train)
model = Model(inputs=baseModel.input, outputs=headModel)
# loop over all layers in the base model and freeze them so they will
# *not* be updated during the first training process
for layer in baseModel.layers:
    layer.trainable = False

The class weights are calculated as:

from sklearn.utils import class_weight
import numpy as np

class_weights = class_weight.compute_class_weight(
               'balanced',
                np.unique(trainGen.classes), 
                trainGen.classes)

These are the class weights:

array([0.18511007, 2.06740331, 1.00321716, 3.53018868, 2.48637874,
       2.27477204, 1.57557895, 6.68214286, 1.04233983, 4.02365591])

and code for training is:

# compile our model (this needs to be done after our setting our layers to being non-trainable
print("[INFO] compiling model...")
opt = SGD(lr=1e-5, momentum=0.8)
model.compile(loss="categorical_crossentropy", optimizer=opt, metrics=["accuracy"])
# train the head of the network for a few epochs (all other layers
# are frozen) -- this will allow the new FC layers to start to become
#initialized with actual "learned" values versus pure random
print("[INFO] training head...")
H = model.fit_generator(
    trainGen,
    steps_per_epoch=totalTrain // BATCH_SIZE,
    validation_data=valGen,
    validation_steps=totalVal // BATCH_SIZE,
    epochs=EPOCHS,
    class_weight=class_weights,
    verbose=1,
    callbacks=callbacks_list)
# reset the testing generator and evaluate the network after
# fine-tuning just the network head