I am training a neural network using Keras. I want to set selected weights in a weight matrix to zero. However, I can't seem to figure out how to do this. Below, I have added a self-contained runnable example of what I want to do:
- I create a simple neural network with 26 inputs and 26 outputs (no hidden layers).
- I add a constraint function to the layers, i.e.,
kernel_constraint. - This calls the function
weight_constraintwhich is supposed to set certain weight matrix values to zero. This should happen inside the__call__function.
I've tried several approaches, but I can't get my mind around how to set the values of the following indices:
row_indices = [1, 2, 3]
col_indices = [10, 11, 12]
to zero in the weight matrix w.
import keras
import numpy
import tensorflow as tf
class weight_constraint(tf.keras.constraints.Constraint):
"""Constrains weight tensors to be centered around `ref_value`."""
def __init__(self):
pass
def __call__(self, w):
shape = w.shape.as_list()
row_indices = [1, 2, 3]
col_indices = [10, 11, 12]
indices = numpy.ravel_multi_index((row_indices, col_indices), dims=shape, order='C')
# Question: how to set the values indicated by "indices" to zero?
return w
full_model = keras.Sequential()
input_layer = keras.layers.Dense(units=26, input_shape=(26,),
activation='tanh',
kernel_initializer='zeros',
bias_initializer='zeros',
kernel_constraint=weight_constraint())
full_model.add(input_layer)
loss = keras.losses.MeanSquaredError()
full_model.compile('RMSprop', loss=loss)
inputs = numpy.random.random((100, 26))
targets = numpy.random.random((100, 26))
training_history = full_model.fit(inputs, targets, epochs=10)