Tensorflow Custom layer weights not training but bias is

user_12899375 0
Viewed 416

I've been writing some custom layers and I have realized my bias values will train but my weights are not training. I'm going to use a very simplified code here to illustrate the issue.

class myWeights(Layer):
    def __init__(self, units, **kwargs): 
        self.units = units
        super(myWeights, self).__init__(**kwargs)      
    def build(self, input_shape):
        self.w = self.add_weight(shape=(input_shape[-1], self.units),
                         initializer='GlorotUniform',
                         trainable=True)
        self.b = self.add_weight(shape=(self.units,),
                         initializer='random_normal',
                         trainable=True)
        super(myWeights, self).build(input_shape)
    def call(self, inputs):
        return tf.matmul(inputs, self.w) + self.b
    def compute_output_shape(self, input_shape):
        return(input_shape[0],self.units)

Now I set up MNIST data to train. I also set a seed so this is reproducible on your end.

tf.random.set_seed(1234)
mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train=tf.keras.utils.normalize(x_train, axis=1)
x_test=tf.keras.utils.normalize(x_test, axis=1)

I build out the model using the functional API

inp=Input(shape=(x_train.shape[1:]))
flat=Flatten()(inp)
hid=myWeights(32)(flat)
out=Dense(10, 'softmax')(hid)
model=Model(inp,out)
model.compile(optimizer='adam',
         loss='sparse_categorical_crossentropy',
         metrics=['accuracy'])

Now when I check the values of the parameters using

print(model.layers[2].get_weights())

I see output like the following, which I have reformatted for easier reading.

  • [array([[ 0.00652369, -0.02321771, 0.01399945, ..., -0.07599965, -0.04356881, -0.0333882 ], [-0.03132245, -0.05264733, 0.05576386, ..., -0.03755575, 0.07358163, -0.02338506], [-0.01808248, 0.04092623, 0.02177643, ..., 0.00971264, 0.07631209, 0.0495184 ], ..., [-0.03780914, 0.00219346, 0.04460619, ..., -0.06703794, 0.03407502, -0.01071112], [-0.0012739 , -0.0683699 , -0.06152753, ..., 0.05373723, 0.03079057, 0.00855774], [ 0.06245673, -0.07649396, 0.06748571, ..., -0.06948434, -0.01416317, -0.08318184]], dtype=float32), *
  • array([ 0.05734033, 0.04822996, 0.04391507, -0.01550511, 0.05383257, 0.05043739, -0.04092903, -0.0081823 , -0.06425817, 0.02402171, -0.00374672, -0.06069579, -0.08422226, 0.02909392, -0.02071654, 0.0422841 , -0.05020861, 0.01267704, 0.0365625 , -0.01743891, -0.01030697, 0.00639807, -0.01493454, 0.03214667, 0.03262959, 0.07799669, 0.05789128, 0.01754347, -0.07558075, 0.0466203 , -0.05332188, 0.00270758], dtype=float32)]*

After training with

model.fit(x_train,y_train, epochs=3, verbose=1)
print(model.layers[2].get_weights())

I find the following output.

  • [array([[ 0.00652369, -0.02321771, 0.01399945, ..., -0.07599965, -0.04356881, -0.0333882 ], [-0.03132245, -0.05264733, 0.05576386, ..., -0.03755575, 0.07358163, -0.02338506], [-0.01808248, 0.04092623, 0.02177643, ..., 0.00971264, 0.07631209, 0.0495184 ], ..., [-0.03780914, 0.00219346, 0.04460619, ..., -0.06703794, 0.03407502, -0.01071112], [-0.0012739 , -0.0683699 , -0.06152753, ..., 0.05373723, 0.03079057, 0.00855774], [ 0.06245673, -0.07649396, 0.06748571, ..., -0.06948434, -0.01416317, -0.08318184]], dtype=float32), *
  • array([-0.250459 , -0.21746232, 0.01250297, 0.00065066, -0.09093136, 0.04943814, -0.13446714, -0.11985168, 0.23259214, -0.14288908, 0.03274751, 0.1462888 , -0.2206902 , 0.14455307, 0.17767513, 0.11378342, -0.22250313, 0.11601174, -0.1855521 , 0.0900097 , 0.21218981, -0.03386492, -0.06818825, 0.34211585, -0.24891953, 0.08827516, 0.2806849 , 0.07634751, -0.32905066, -0.1860122 , 0.06170518, -0.20212872], dtype=float32)]*

I can see that the bias values have changed but the weight values are static. I'm not sure at all why this is occurring.

tensorflow2.0 keras python
1 Answers

What your trying is Multilayer Perceptron (MLP), MLP is usually composed of one(passthrough) input layer, one or more layers of TLUs, called hidden layers, and one final layer of TLUs called the output layer.

Here the signal flows only in one direction (from the inputs to the outputs), so this architecture is an example of a feedforward neural network (FNN).

See this link which will explain feedforward neural network.

Coming to the explanation of your code, you are initializing weights using some initializers. So the first initialization of weights happens at the hidden layer and then gets updated in the next Dense layer.
So whatever the weights are initialized will remain the same even after training in the hidden layer since it is a feedforward neural network means it is not dependent on the output of the current layer.

But if you want to check your code then you can include one more hidden layer exactly as the one which is present and see the weights for layer 3(hidden layer 2) which looks something like this. 

inp=Input(shape=(x_train.shape[1:]))
flat=Flatten()(inp)
hid=myWeights(32)(flat)
hid2=myWeights(32)(hid)
out=Dense(10, 'softmax')(hid2)
model=Model(inp,out)
model.compile(optimizer='adam',
         loss='sparse_categorical_crossentropy',
         metrics=['accuracy'])

Then by printing the weights before fit and after fit for hidden2 layer will give you different weights, since the weights for the hidden 2 layer is dependent on the output of the hidden 1 layer.

print(model.layers[3].get_weights())
user_134652580
Related