Tensorflow: How to apply a regularizer on a tensor?

user_17142635 0
Viewed 264

I am implementing a model in Tensorflow 2, and I want to apply a penalization on a tensor (multiplication from two layers' outputs) in my model. I am used to use regularization on layers (kernel, bias or activity regularization).

I could build a custom layer that only has an activity regularization, but I am hopping that there is a simpler solution to add regularization to a tensor.

I saw this code in Tensorflow:

regularizer = tf.keras.regularizers.L2(2.)
tensor = tf.ones(shape=(5, 5))
regularizer(tensor)

Which outputs:

<tf.Tensor: shape=(), dtype=float32, numpy=50.0>

But does this only compute the regularization value or it also add it to my model's loss?

Or would adding self.add_loss(tf.keras.regularizers.L2(2.)(tensor)) in my call function work?

How would you add a penalty on a tensor?

It is my first question on stackoverflow so sorry if I didn't ask in the good place.

regularized tensor keras tensorflow
1 Answers

No, the regularizer classes pretty much only handle the computation of the penalty itself. E.g. the source code for the L2 regularizer has

def __call__(self, x):
    return self.l2 * tf.reduce_sum(tf.square(x))

and there is also nothing in __init__ that points at any "housekeeping" work. However, activity_regularizer is actually an argument for the base Layer class so any subclassed layer should be able to handle it by default, meaning it should be easy to write a custom layer (you essentially only have to write the call method which sounds like a one-liner in your case).

You may even be able to use a Lambda layer to avoid having to write any sub-classing code, since these also inherit from Layer. However, the docs mention that there are some issues with Lambda regarding saving and loading models...

user_93931020
Related