what is the purpose and intuition of Layer.add_loss()? How does it work? Does it calculate the loss for the current layer only?

user_11725056 0
Viewed 70

I was looking at the tensorflow tutorial about making a custom layer and found a code which is something like:

# A layer that creates an activity regularization loss
class ActivityRegularizationLayer(keras.layers.Layer):
    def __init__(self, rate=1e-2):
        super(ActivityRegularizationLayer, self).__init__()
        self.rate = rate

    def call(self, inputs):
        self.add_loss(self.rate * tf.reduce_sum(inputs))
        return inputs

and then they have used the nested Layers as

class OuterLayer(keras.layers.Layer):
    def __init__(self):
        super(OuterLayer, self).__init__()
        self.activity_reg = ActivityRegularizationLayer(1e-2)

    def call(self, inputs):
        return self.activity_reg(inputs)

I know the outer layer will address the weights, biases and losses of the inner layer too but what I fail to understand is that why is it not a custom loss function but a layer? Is this supposed to be collecting the loss for the specific layer only? If yes, I don't see a way out about what is it doing and how is it?

tensorflow2.0 keras tensorflow deep-learning neural-network
1 Answers

You can ignore the OuterLayer - that was demonstrating the losses attribute is inclusive of losses in embedded layers. Keras does a lot of tricks to make this kind of stuff work.

The ActivityRegularizer shows an example where you are not regularizing the weight or bias. Instead, you are adding a loss to the output of the layer - aka the activity. You would use an ActivityRegularizer when the previous layer doesn't already have some built-in way to regularize the activity. For example a tf.keras.layers.Dense layer has an activity regularizer built in, so you wouldn't use the ActivityRegularizer layer. But ReLU does not, so if you wanted to regularize the activity of a ReLU layer, you'd add a ActivityRegularizer layer after the ReLU.

user_106487650
Related