decide whether to update the weights based on a condition on features

user_12338521 0
Viewed 103

I would like to decide if to train some specific weight, based on feature values.

As a simple attempt I would like to "touch" only the weights between input and 1st layer such that if a condition on a feature i holds, then DO NOT train (update) all those weights connecting the 1st layer with input neuron i

Let us say I have model like this:

model = keras.Sequential([
layers.Dense(2, activation=tf.nn.relu, input_shape=(N, 4)),
layers.Dense(1, activation=tf.nn.relu)])

I want everything working normally if I have inputs as [1,2,3,4] with all nonzero values.But, then I would like to provide an example like [1,2,3,0] and make the weights related the 4th neuron of the input layers to stay the same and not updated.I'm trying to simulate a "missing feature" behaviour

backpropagation keras tensorflow neural-network python
1 Answers

I might be completely wrong. But I think this question makes no mathematical sense. If the input is zero, the gradients will be zero. And so no update to the weights will be performed. Let's make the test.

Let's make some input with a zero:

import tensorflow as tf
import numpy as np

x = np.array([[1.2, 1.1, 0., 2.3, 4.1]], dtype=np.float32)
y = [0.]

As you can see, the third value is missing.

Now let's make a forward pass with a random target, and look at the gradients:

x = np.array([[1.2, 1.1, 0., 2.3, 4.1]], dtype=np.float32)
y = [0.]

model = tf.keras.Sequential([
    tf.keras.layers.Dense(8, activation='relu'),
    tf.keras.layers.Dense(3, activation='softmax')])

with tf.GradientTape() as tape:
    out = model(x)
    loss = tf.losses.sparse_categorical_crossentropy(y, out)
    gradients = tape.gradient(loss, model.trainable_variables)

np.round(gradients[0].numpy().tolist(), 2)

array([[0.  , 0.  , 0.  , 0.3 , 0.  , 0.  , 0.17, 1.29],
       [0.  , 0.  , 0.  , 0.28, 0.  , 0.  , 0.15, 1.18],
       [0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  ],
       [0.  , 0.  , 0.  , 0.58, 0.  , 0.  , 0.32, 2.47],
       [0.  , 0.  , 0.  , 1.03, 0.  , 0.  , 0.57, 4.4 ]])

All the gradients related to the 3rd column of the input (row 3) are zero. When optimizer.apply_gradients will be applied in model.fit, nothing will be performed on the weights related to the missing value because the gradients are zero.

user_109083750
Related