How to create a custom loss function in Keras that evaluates prediction after each epoch?

user_12187261 0
Viewed 692

I'm working on a neural network in Keras that translates English sentences into a custom language. For this, I'd like to create a custom loss function that takes the prediction for each sentence and evaluates whether it complies with the grammar rules of the custom language and if not adds value to the standard loss function.

How can I evaluate a tensor after each epoch but not during compilation?

Below is my custom loss function. As during compilation of the model there is no batch yet, y_pred has the shape (None, x, y) and can't be evaluated to get the prediction. My idea to circumvent this was to assign a standard loss function during compilation and when batches arrive calculate the custom loss. Unfortunately the custom loss is never reached.

def custom_loss(tokenizer, punishment_rate):

    def compile_loss(y_true, y_pred):
        shape = K.int_shape(y_pred)
        #standard loss function
        loss = K.sparse_categorical_crossentropy(y_true, y_pred)

        if shape[0] is not None:
            #THIS is never reached and that's the problem
            prediction = logits_to_text(K.eval(y_pred), tokenizer)

            #test if prediction complies to grammar rules
            compileable = compiles(prediction) ^ 1

            compile_error = compileable * punishment_rate
            loss =  K.sparse_categorical_crossentropy(y_true, y_pred, axis=-1) * (1 + compile_error)

        return loss

    return compile_loss

Is there any workaround for evaluating a tensor only when it was filled with a batch? Or alternatively, change the loss function after compilation of the model via a callback without it having to recompile the model?

loss-function keras machine-learning nlp python-3.x
1 Answers

As per keras source, you can use a Loss Function Wrapper to create a Custom Loss Function class and then pass it to your model seamlessly.

As an example:

#Import the wrapper
from keras.losses import LossFunctionWrapper

#Create your class extending the wrapper
class MyLossFunction(LossFunctionWrapper):

#Implement the constructor - here you can give extended arguments to it.
   def __init__(self, 
                 tokenizer, 
                 punishment_rate,
                 reduction=losses_utils.Reduction.SUM_OVER_BATCH_SIZE,
                 name='my_custom_text_function'):
        super(MyLossFunction, self).__init__(
            my_function, 
            name=name,
            reduction=reduction,
            tokenizer = tokenizer, 
            punishment_rate= punishment_rate)

#Now you have to define your function "my_function":
#Please, notice that ALL loss functions that follow keras model needs two arguments: 
#y_true (correct result) and y_pred (the result obtained in the network).
def my_function(y_true, y_pred, tokenizer, punishment_rate):
   shape = K.int_shape(y_pred)
   if shape[0] is not None:
       prediction = logits_to_text(K.eval(y_pred), tokenizer)

       #test if prediction complies to grammar rules
       compileable = compiles(prediction) ^ 1

       compile_error = compileable * punishment_rate
       return K.sparse_categorical_crossentropy(y_true, y_pred, axis=-1) * (1 + compile_error)
   return K.sparse_categorical_crossentropy(y_true, y_pred)

You can then instantiate it and use in your compiler:

custom_loss= MyLossFunction(tokenizer = ..., punishment_rate = ...)
classifier.compile(optimizer=optimizer, 
                      loss=custom_loss, 
                      metrics= ['binary_accuracy'])
user_32880040
Related