Bounding Box regression using Keras transfer learning gives 0% accuracy. The output layer with Sigmoid activation only outputs 0 or 1

I am trying to create an object localization model to detect license plate in an image of a car. I used VGG16 model and excluded the top layer to add my own dense layers, with the final layer having 4 nodes and sigmoid activation to get (xmin, ymin, xmax, ymax).

I used the functions provided by keras to read image, and resize it to (224, 244, 3), and also used preprocess_input() function to process the input. I also tried to manually process the image by resizing with padding to maintain proportion, and normalize the input by dividing by 255.

Nothing seems to work when I train. I get 0% train and test accuracy. Below is my code for this model.

def get_custom(output_size, optimizer, loss):

    vgg = VGG16(weights="imagenet", include_top=False, input_tensor=Input(shape=IMG_DIMS))

    vgg.trainable = False

    flatten = vgg.output
    flatten = Flatten()(flatten)

    bboxHead = Dense(128, activation="relu")(flatten)
    bboxHead = Dense(32, activation="relu")(bboxHead)

    bboxHead = Dense(output_size, activation="sigmoid")(bboxHead)

    model = Model(inputs=vgg.input, outputs=bboxHead)
    model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])

    return model

X and y were of shapes (616, 224, 224, 3) and (616, 4) respectively. I divided the coordinates by the length of the respective sides so each value in y is in range (0,1).

I'll link my python notebook below from github so you can see the full code. I am using google colab to train the model. https://github.com/gauthamramesh3110/image_processing_scripts/blob/main/License_Plate_Detection.ipynb

Thanks in advance. I am really in need of help here.