Triplet Loss model not getting any result Siamese Model

I have been trying to build a Siamese model using the Triplet Loss addon by Tensorflow (this tries to, working in a batch of anchor, positive(same person) and negative(other person), minimize the distance between faces of identical identity and maximizing distance between faces of didferent people.

I started training a whole CNN from scratch and then I tried to do the method transfer learning. In both I could not reach my aim, distancing two faces or make it close in the case of two faces of the same person using the embedding of each model.

What I basically do, for example, in transfer learning is this (in the case of CNN from scratch is just building a model typically a Inception based):

MODEL_URL = "https://tfhub.dev/google/imagenet/resnet_v2_50/feature_vector/5"
enc_model = tf.keras.Sequential([
    tf.keras.layers.Rescaling(scale=1. / 255),
    tf.keras.layers.RandomFlip("horizontal"),
    tf.keras.layers.RandomRotation(0.2),
    hub.KerasLayer(MODEL_URL, trainable=True),
    tf.keras.layers.Dropout(0.3),
    tf.keras.layers.Dense(384, activation=None),  # No activation on final dense layer
    tf.keras.layers.Lambda(lambda x: tf.math.l2_normalize(x, axis=1))  # L2 normalize embeddings
])
enc_model.build([None, 100, 100, 3])    
enc_model.compile(
      optimizer=tf.keras.optimizers.Adam(learning_rate=0.0001),
      loss=tfa.losses.TripletSemiHardLoss())

directory_test='Database/PubFig+CelebData_faces/'
train_ds = balanced_image_dataset_from_directory(
    directory_test, num_classes_per_batch=16,
    num_images_per_class=8, image_size=(100, 100),
    seed=555,validation_split=0.2, subset='training',
    safe_triplet=True)
val_ds = balanced_image_dataset_from_directory(
    directory_test, num_classes_per_batch=16,
    num_images_per_class=8, image_size=(100, 100),
    seed=555,validation_split=0.2, subset='validation',
    safe_triplet=True)

siamese_history = enc_model.fit(
    train_ds,
    validation_data=val_ds,
    epochs=epochs,
    callbacks=callbacks,
    verbose=1)

Some lines I have skipped because they are not relevant to the case we are dealing with. The dataset is this one: https://www.kaggle.com/datasets/kaustubhchaudhari/pubfig-dataset-256x256-jpg I crop the faces and rescaled them to 100x100 before training. Then the augmentation is done inside the model.

What balanced_image_dataset_from_directory does is create a tf.dataset with a batch equally distributed of n classes(people) with m images of their faces.

The results that I get are this: Transfer learning embeddings before training

Transfer learning embeddings after training

Transfer learning Training

I recognize the overfitting in fine tuning, but I have tried many things and I have not came up with any result. I hope you could help me.