Neural Network isn't learning anything meaningful using Triplet Loss

I am working on a triplet loss based model for this Kaggle competition.

Short Description- In this competition, we have been challenged to build an algorithm to identify individual whales in images by analyzing a database of containing more than 25,000 images, gathered from research institutions and public contributors.

https://www.kaggle.com/c/humpback-whale-identification?rvi=1

I have decided to use a Siamese network architecture and train it to give me encodings which I can then use to calculate the distance between two pictures of whales. If this distance is below a particular threshold the two pictures belong to the same whale and if this distance is greater then, they aren't the same whale.

This is the Triplet loss function(learnt it from Andrew's deeplearning specialization) I used but i also normalized the encoding's to make the loss function more interpretable(easier to determine margin and split point) across different models(if that makes sense).(First, tried it without the normalization and when it didnt work i tried normalizing.) I also have tried changing alpha(margin) and varied it from 0.2 to 0.6.

from tensorflow.nn import l2_normalize as norm_l2

def triplet_loss(y_true, y_pred, alpha = 0.3):
    """
    Arguments:
    y_true -- true labels, required when you define a loss in Keras, you don't need it in this function.
    y_pred -- python list containing three objects:
            anchor -- the encodings for the anchor images, of shape (None, 128)
            positive -- the encodings for the positive images, of shape (None, 128)
            negative -- the encodings for the negative images, of shape (None, 128)
    
    Returns:
    loss -- real number, value of the loss
    """
    
    anchor, positive, negative = y_pred[0], y_pred[1], y_pred[2]
    anchor, positive, negative = norm_l2(anchor), norm_l2(positive), norm_l2(negative)

    # Step 1: Compute the (encoding) distance between the anchor and the positive
    pos_dist = tf.reduce_sum(tf.square(tf.subtract(anchor,positive)), axis = -1)
    # Step 2: Compute the (encoding) distance between the anchor and the negative
    neg_dist = tf.reduce_sum(tf.square(tf.subtract(anchor,negative)), axis = -1)
    # Step 3: subtract the two previous distances and add alpha.
    basic_loss = tf.add(tf.subtract(pos_dist, neg_dist), alpha)
    # Step 4: Take the maximum of basic_loss and 0.0. Sum over the training examples.
    loss = tf.reduce_sum(tf.maximum(basic_loss, 0.0))
  
    return loss

This is an example of one of the model architectures i tried out. I have tried using pretrained Facenet, ResNet, DenseNet and Xception till now. I have tried Freezing different numbers of layers in each.

R = tf.keras.applications.ResNet50(include_top=False, weights = 'imagenet', input_shape=(224,224,3))

lr = 0.0001
optimizer = Adam(learning_rate=lr)
R.compile(optimizer=optimizer, loss = triplet_loss)

for layer in R.layers[0:30]:
    layer.trainable = False

em_Rmodel = Sequential([
                          R,
                          GlobalAveragePooling2D(),
                          #tf.keras.layers.GlobalMaxPooling2D(),
                          Dense(512, activation='relu'),
                          bn(),
                          Dense(256, activation = 'sigmoid'),
                          Dense(128, activation = 'sigmoid')
                          ])

def make_tripletModel(model):

    #I was manually changing the input shape to fit the default shape of pretrained networks
    A = Input(shape = (224, 224, 3), name='anchor')
    P = Input(shape = (224, 224, 3), name = 'anchorPositive')
    N = Input(shape = (224, 224, 3), name = 'anchorNegative')

    enc_A = model(A)
    enc_P = model(P)
    enc_N = model(N)

    tripletModel = Model(inputs=[A, P, N], outputs=[enc_A, enc_P, enc_N])
    return tripletModel

tripletModel = make_tripletModel(em_Rmodel)

I have been training using semi-hard triplets and have also been augmenting data properly to generate more training images.

This is the batch generator that i used for training. crop_batch is a function that crops images to show only the whale's tail, using which one can identify whales. It uses a DenseNet trained on more than 1000 images with whale tails and the bounding box surrounding it. Does the work sufficiently well.

def batch_generator_RN(batch_size = batch_size, ishape = (256, 256, 3), model_input_shape = (224, 224, 3)):
    triplet_generator = get_triplets()
    y_val = np.zeros((batch_size, 2, 1))
    anchors = np.zeros((batch_size, ishape[0], ishape[1], ishape[2]))
    positives = np.zeros((batch_size, ishape[0], ishape[1], ishape[2]))
    negatives = np.zeros((batch_size, ishape[0], ishape[1], ishape[2]))

    while True:        
        for i in range(batch_size):
            anchors[i], positives[i], negatives[i] = next(triplet_generator)
        
        anc = crop_batch(anchors, batch_size= batch_size, img_shape=model_input_shape)
        pos = crop_batch(positives, batch_size= batch_size, img_shape=model_input_shape)
        neg = crop_batch(negatives, batch_size= batch_size, img_shape=model_input_shape)

        x_data = {'anchor': anc,
                  'anchorPositive': pos,
                  'anchorNegative': neg
                  }

        yield (x_data, [y_val, y_val, y_val])

And finally, this, in general, is how i have been trying to train these models. I have tried reducing and increasing learning rate, batch_size = 16.

lr = 0.0001
optimizer = Adam(learning_rate=lr)
tripletModel.compile(optimizer = optimizer, loss = triplet_loss)


es = EarlyStopping(monitor='loss', patience=20, min_delta=0.05, restore_best_weights=True)
#mc = ModelCheckpoint('Rmodel.h5', monitor='loss', save_best_only=True, save_weights_only=True)
rlr = ReduceLROnPlateau(monitor='loss',min_delta=0.05,factor = 0.1,patience = 5, verbose = 1, min_lr = 0)

gen = batch_generator(batch_size)
tripletModel.fit(gen, steps_per_epoch=64, epochs = 40, callbacks=[es, rlr])

So after training all these models, in some models the triplet loss does go down for a while but then plateaus and basically learns nothing meaningful(which basically means that just by looking at the distance between two embeddings i cant figure out if they are the same whale or not.). In other models, immediately after the first or the second epoch the weights converge and don't change at all and doesn't learning anything.

I have tried a very wide range of learning rates and i am pretty sure that it isnt the problem.

Please tell me if i should add all the code files for you to understand the problem better. The reason i havent done it yet because i havent cleaned it but will gladly do so if required. Thanks.