How to create the custom loss function by adding negative entropy to the cross-entropy?

user_7575552 0
Viewed 467

I recently read a paper entitled "REGULARIZING NEURAL NETWORKS BY PENALIZING CONFIDENT OUTPUT DISTRIBUTIONS https://arxiv.org/abs/1701.06548". The authors discuss regularizing neural networks by penalizing low entropy output distributions through adding a negative entropy term to the negative log-likelihood and creating a custom loss function for model training.

enter image description here

The value β controls the strength of confidence penalty. I have written a custom function for categorical cross-entropy as shown below but the negative entropy term need to be added to the loss function.

import tensorflow as tf
def custom_loss(y_true, y_pred):
    cce = tf.keras.losses.CategoricalCrossentropy()
    cce_loss = cce(y_true, y_pred)    
    return cce_loss
keras tensorflow deep-learning numpy python
2 Answers

You do not need a custom loss, as it can be implemented as an activity regularizer (one applied to the output of a layer):

def regularizer(beta):
    def entropy_reg(inp):
        return -beta * K.mean(inp * K.log(inp))

Then this can be applied to your output layer:

model = Sequential()
#Add layers here
model.add(Dense(num_classes, activation="softmax",
          activity_regularizer=regularizer(0.01)))
user_3491300

The entropy of y_pred is essentially the categorical cross entropy between y_pred and itself:

enter image description here

def custom_loss(y_true, y_pred, beta):
    cce = tf.keras.losses.CategoricalCrossentropy()
    return cce(y_true, y_pred) - beta*cce(y_pred, y_pred)
user_63313690
Related