I am trying to replace all ReLU activation functions in the MobileNetV2 with some custom activation functions(abs, swish, leaky relu etc.). First I liked to start with abs. I saw a few similar posts but they were not really helpful for my problem -> see this one.
backbone = tf.keras.applications.mobilenet_v2.MobileNetV2(
input_shape=IN_SHAPE, include_top=False, weights=None)
x = tf.keras.layers.GlobalAveragePooling2D()(backbone.output)
x = tf.keras.layers.Dropout(dropout_rate)(x)
kernel_initializer = tf.random_normal_initializer(mean=0.0, stddev=0.02)
bias_initializer = tf.constant_initializer(value=0.0)
x = tf.keras.layers.Dense(
NUM_CLASSES, activation='sigmoid', name='Logits',
kernel_initializer=kernel_initializer,
bias_initializer=bias_initializer)(x)
model = tf.keras.models.Model(inputs=backbone.input, outputs=x)
for layer in model.layers:
layer.trainable = True
smooth = 0.1
loss = tf.keras.losses.BinaryCrossentropy(label_smoothing=smooth)
model = replace_relu_with_abs(model) # function I like to call to replace the activation function
model.compile(
optimizer=optimizer,
loss=loss,
metrics=['accuracy'])
model.save("mobilenetv2-abs")
model = tf.keras.models.load_model("mobilenetv2-abs")
print(model.summary())
# Here the function I like to implement
def replace_relu_with_abs(model):
for layer in model.layers:
# do something
return model
I tried to implement the solution from the link above but that didn't work and after debugging it I saw that MobileNetV2 has in the Conv2d layer linear as an activation function and the relu activation function is a separate layer which comes after the BatchNormalization layer.
Has anyone a tip or solution how to replace the relu activation functions with custom ones( here its rather a ReLU activation layers)?
I am using tensorflow version 2.6