Keras - using activation function with a parameter

user_4170533 0
Viewed 6698

How is it possible to use leaky ReLUs in the newest version of keras? Function relu() accepts an optional parameter 'alpha', that is responsible for the negative slope, but I cannot figure out how to pass ths paramtere when constructing a layer.

This line is how I tried to do it,

model.add(Activation(relu(alpha=0.1))

but then I get the error

TypeError: relu() missing 1 required positional argument: 'x'

How can I use a leaky ReLU, or any other activation function with some parameter?

keras
4 Answers

You can build a wrapper for parameterized activations functions. I've found this useful and more intuitive.

class activation_wrapper(object):
    def __init__(self, func):
        self.func = func

    def __call__(self, *args, **kwargs):
        def _func(x):
            return self.func(x, *args, **kwargs)
        return _func

Of course I could have used a lambda expression in call. Then 

wrapped_relu = activation_wrapper(relu).

Then use it as you have above

model.add(Activation(wrapped_relu(alpha=0.1))

You can also use it as part of a layer

model.add(Dense(64, activation=wrapped_relu(alpha=0.1))

While this solution is a little more complicated than the one offered by @Thomas Jungblut, the wrapper class can be reused for any parameterized activation function. In fact, I used it whenever I have a family of activation functions that are parameterized.

user_119200010

Keras defines separate activation layers for the most common use cases, including LeakyReLU, ThresholdReLU, ReLU (which is a generic version that supports all ReLU parameters), among others. See the full documentation here: https://keras.io/api/layers/activation_layers

Example usage with the Sequential model:

import tensorflow as tf
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.InputLayer(10))
model.add(tf.keras.layers.Dense(16))
model.add(tf.keras.layers.LeakyReLU(0.2))
model.add(tf.keras.layers.Dense(1))
model.add(tf.keras.layers.Activation(tf.keras.activations.sigmoid))
model.compile('adam', 'binary_crossentropy')

If the activation parameter you want to use is unavailable as a predefined class, you could use a plain lambda expression as suggested by @Thomas Jungblut:

from tensorflow.keras.layers import Activation
model.add(Activation(lambda x: tf.keras.activations.relu(x, alpha=0.2)))

However, as noted by @leenremm in the comments, this fails when trying to save or load the model. As suggested you could use the Lambda layer as follows:

from tensorflow.keras.layers import Activation, Lambda
model.add(Activation(Lambda(lambda x: tf.keras.activations.relu(x, alpha=0.2))))

However, the Lambda documentation includes the following warning:

WARNING: tf.keras.layers.Lambda layers have (de)serialization limitations!

The main reason to subclass tf.keras.layers.Layer instead of using a Lambda layer is saving and inspecting a Model. Lambda layers are saved by serializing the Python bytecode, which is fundamentally non-portable. They should only be loaded in the same environment where they were saved. Subclassed layers can be saved in a more portable way by overriding their get_config method. Models that rely on subclassed Layers are also often easier to visualize and reason about.

As such, the best method for activations not already provided by a layer is to subclass tf.keras.layers.Layer instead. This should not be confused with subclassing object and overriding __call__ as done in @Anonymous Geometer's answer, which is the same as using a lambda without the Lambda layer.

Since my use case is covered by the provided layer classes, I'll leave it up to the reader to implement this method. I am making this answer a community wiki in the event anyone would like to provide an example below.

user_138229430
Related