In the TensorFlow documentation it is highlighted that it is important during fine tuning to set the base_model to ’inference mode’ setting the parameter training = False when calling the base_model. The reason to do so is because of the tf.keras.layers.BatchNormalization layers, that should be executed in inference mode during fine tuning.
TensorFlow documentation on Fine Tuning
But setting the base_model to inference mode will also affect the tf.keras.layers.Dropout in the base_model as these will then also run in inference mode and will not apply any dropout at all.
What is useful for getting meaningful results when fine tuning a model?
Running the dropout layers in the base_model in inference mode (no dropout at all) or running them in training mode applying the dropout as defined in the base_model?