How to write a conditonal regressional loss function based on binary cross entropy loss for Keras model

Question

I am building a key-point detection system of the human face. The goal is to have an image of the face be input into the model, and the model then detects anatomical landmarks in the image (eyes, nose) and outputs the pixel coordinates of the landmarks that are visible. There are three targets per landmark: x, y, visible. X and Y are the pixel coordinates, and visible is whether the landmark is in the image or not. The plan is to first have a binary cross entropy loss between predicted visibility and true visibility. Then, the second loss is a regression loss (I'm using MAPE) between the x,y coordinates and the targets. However, the regression loss would only be calculated for landmarks that are visible. The loss would look something like:

#Pseudo-code
def loss(y_true,y_pred):
    if y_true[2] == 1
       #Probability that landmark is in image
       #Compute binary cross entropy loss
       #Compute MAPE regression loss
       Total_loss = Binary_loss + MAPE_loss
       return Total_loss

    else:
       Total_loss = Binary loss
       return Total_loss

Once the loss function is written, how would I go about implementing it in code? I know how to create models for each problem (checking the coordinates, and separately checking the visibility), but I'm not sure exactly how to go about combining the two heads with the conditional loss function. How would I combine the layers (Conv, Flatten, Dense for each head) to get the desired output? Thank you!

EDIT: I'm not able to upload the data, but here is an image of it. The first 9 columns are the coordinates, and visibility of the landmarks. The last column is the corresponding image which has been flattened. When I load in the data for training, these are the steps I do:

###Read in data file
file = "Directory/file.csv"
train_data = pd.read_csv(file)
###Convert each coordinate column to type float64
train_data['xreye'] = train_data['xreye'].astype(np.float64)
...
###Convert image column to string type
train_data['Image'] = train_data['Image'].astype(str)

#Image is feature, other values are labels to predict later
#Image column values are strings, also some missing values, have to split
##string by space and append it and handle missing values
imag = []
for i in range(len(train_data)):
    img = train_data['Image'][i].split(' ')
    img = ['0' if x == '' else x for x in img]      
    imag.append(img) 
#Reshape and convert to float value
image_list = np.array(imag,dtype = 'uint8')
X_train = image_list.reshape(-1,256,256,1)

####Get pixel coordinates and visibility targets
training = train_data[['xreye','yreye','reyev','xleye','yleye','leyev','xtsept','ytsept','tseptv']]
y_train = []
for i in range(len(train_data)):
    y = training.iloc[i,:]
    y_train.append(y)

y_train = np.array(y_train, dtype='float')

EDIT: Model code, loss function, and fit method.

###Loss function
visuals_mask = [False, False, True] * 3
def loss_func(y_true, y_pred):
    visuals_true = tf.boolean_mask(y_true, visuals_mask, axis=1)
    visuals_pred = tf.boolean_mask(y_pred, visuals_mask, axis=1)
    visuals_loss = tf.keras.losses.BinaryCrossentropy(visuals_true, visuals_pred)
    visuals_loss = tf.reduce_mean(visuals_loss)

    coords_true = tf.boolean_mask(y_true, ~np.array(visuals_mask), axis=1)
    coords_pred = tf.boolean_mask(y_pred, ~np.array(visuals_mask), axis=1)
    coords_loss = tf.keras.losses.MeanAbsolutePercentageError(coords_true, coords_pred)
    coords_loss = tf.reduce_mean(coords_loss)

    return coords_loss + visuals_loss
####Model code
model = Sequential()

model.add(Conv2D(32, (3,3), activation='relu', padding='same', use_bias=False, input_shape=(256,256,1)))
model.add(BatchNormalization())
model.add(MaxPool2D(pool_size=(2,2)))

model.add(Conv2D(64, (3,3), activation='relu', padding='same', use_bias=False))
model.add(BatchNormalization())
model.add(MaxPool2D(pool_size=(2,2)))

model.add(Conv2D(128, (3,3), activation='relu', padding='same', use_bias=False))
model.add(BatchNormalization())
model.add(MaxPool2D(pool_size=(2,2)))

model.add(Flatten())

model.add(Dense(128, activation='relu'))
model.add(Dense(64, activation='relu'))
model.add(Dense(32, activation='relu'))
model.add(Dense(16, activation='relu'))
model.add(Dense(8, activation='relu'))
model.add(Dense(4, activation='relu'))
model.add(Dense(9, activation='linear'))
model.summary()
model.compile(optimizer='adam', loss=loss_func)

###Model fit
checkpointer = ModelCheckpoint('C:/Users/Cloud/.spyder-py3/x_y_shift/weights/vis_coords_TEST.hdf5', monitor='val_loss', verbose=1, mode = 'min', save_best_only=True)
out = model.fit(X_train,y_train,epochs=5,batch_size=4,validation_split=0.1, verbose=1, callbacks=[checkpointer])

user_6402099 · Answer

I can't be sure beacuse I don't have data to reproduce the problem but these are the steps in my head:

Use boolean masking to get the 2, 5 and 8. indexes from the output:

visuals_mask_ = [False, False, True] * 3

# in the loss function
visuals_true = tf.boolean_mask(y_true, visuals_mask_, axis=-1) # do the same with preds

Compute the loss for the visuals

visuals_loss = binary_crossentropy(visuals_true, visuals_pred) # use sparse if that's the case

Get the coordinates' outputs just like we did for visuals but with reversed visuals_mask. I believe tf.boolean_mask(y_true, tf.math.logical_not(visuals_mask_, axis=-1)) should work.
Compute MAPE for the rest (coords_true and coords_pred)
Get the means for both losses by tf.reduce_mean
Get sum of losses and return it

I hope these will provide some insight.

Edit: I tried the following and seems like it's working:

y_true = tf.convert_to_tensor(np.random.rand(32, 9))
y_pred = tf.convert_to_tensor(np.random.rand(32, 9))

visuals_mask = [False, False, True] * 3

def loss_func(y_true, y_pred):
    visuals_true = tf.boolean_mask(y_true, visuals_mask, axis=1)
    visuals_pred = tf.boolean_mask(y_pred, visuals_mask, axis=1)
    visuals_loss = binary_crossentropy(visuals_true, visuals_pred)
    visuals_loss = tf.reduce_mean(visuals_loss)

    coords_true = tf.boolean_mask(y_true, ~np.array(visuals_mask), axis=1)
    coords_pred = tf.boolean_mask(y_pred, ~np.array(visuals_mask), axis=1)
    coords_loss = mean_absolute_percentage_error(coords_true, coords_pred)
    coords_loss = tf.reduce_mean(coords_loss)

    return coords_loss + visuals_loss

loss_func(y_true, y_pred)

What I assumed here is:

Your output has actually has length of 9 ((batch_size, 9)).
Custom loss calculations may differ in this demonstration and actual training because of eager execution.

Edit 2: I tried it with this kind of model and it's seems to work:

model = Sequential()

model.add(Conv2D(4, 10, data_format='channels_last', input_shape=(256, 256, 1)))
model.add(Flatten())
model.add(Dense(9, activation='sigmoid'))

model.compile('adam', loss=loss_func)

How to write a conditonal regressional loss function based on binary cross entropy loss for Keras model

1 Answers