Listwrapper not allowing multiplication of learning rate and thus no update of weight for Nueral Network

user_19152216 0
Viewed 66

I am new to tensorflow and nueral networks. I am trying to create a NN to estimate y = x^2

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt


x_train = tf.constant(value = np.linspace(-10,10,50),dtype='float32')
x_train = tf.reshape(x_train,shape=[50,1])
y_train = x_train**2
layers = [1,3,4,1]

I created a nueral network class to obtain my weights and biases and run forward propagation.

class NN(tf.Module):
        def __init__(self,layers,name=None):
            super().__init__(name=name)
            self.layers = layers
    
            self.weights, self.biases = self.initialze(layers)
            
        def initialze(self,layers) :
            num_layers = len(layers)
            weights = []
            biases = []
            for i in range(num_layers-1):
                in_dim = layers[i]
                out_dim = layers[i+1]
                stddev = np.sqrt(2/(in_dim + out_dim))
                b = tf.Variable(tf.zeros([1,layers[i+1]], dtype='float32'), dtype='float32')
                W =  tf.Variable(tf.random.truncated_normal([in_dim, out_dim], stddev=stddev), dtype='float32')
                weights.append(W)
                biases.append(b)
            return weights, biases
        def __call__(self,x):
            Z = x
            num_layers = len(self.layers)
            for i in range(num_layers-1):
                Z =tf.math.add(tf.linalg.matmul(Z ,self.weights[i]),self.biases[i])
            return Z
            
                
    My_NN = NN(layers)

Next I created a class updat to do backward propogation

class updat:
    def __init__(self,y_train,x_train):
        
        self.y_train =  y_train
        self.x_train = x_train
        self.l_r = 0.1
    def get_grad(self,My_NN):
            with tf.GradientTape(persistent=True) as tape:
                tape.watch(My_NN.weights)
                tape.watch(My_NN.biases)
                loss = tf.reduce_mean(tf.square(self.y_train-My_NN(self.x_train)))
                
            dw,db = tape.gradient(loss, [My_NN.weights,My_NN.biases])
            print(dw,'weight')
            print(db,'biases')
            My_NN.weights -= (self.l_r * dw)
            My_NN.biases -=(self.l_r * db)
            
            del tape

            return loss
    def report(self, loss):
        return f"W = {My_NN.weights.numpy():1.2f}, b = {My_NN.biases.numpy():1.2f}, loss={loss:2.5f}"
   

        
    def prop(self,epochs,My_NN):
        for epoch in epochs:
            loss = self.get_grad(My_NN)
            
            current_loss = loss
           
            print(f"Epoch {epoch:2d}:")
            print("    ", report(current_loss,My_NN))

But when I run the code

model = updat(y_train,x_train)
epochs = range(10)
model.prop(epochs,My_NN)

I get an error saying

My_NN.weights -= (self.l_r * dw)

My_NN.biases -=(self.l_r * db)

TypeError: can't multiply sequence by non-int of type 'float'

I tried substituting My_NN.weights -= (lr*dw) with My_NN.weights.assign_sub(lr*dw) still it shows that

'ListWrapper' object has no attribute 'assign_sub'

Is there any solution for this?

tensorflow2.0 tensorflow python
1 Answers

TURN

My_NN.weights -= (self.l_r * dw)
My_NN.biases -=(self.l_r * db)

TO

for weight,d_weight in zip(My_NN.weights,dw):
    weight.assign_sub(self.l_r * d_weight) 
for bias,d_bias in zip(My_NN.biases,db):
    bias.assign_sub(self.l_r * d_bias)

can solve the problem.

Because My_NN.weights is a list of tf.Variable's ref and dw is corresponding list of tf.constant. We cannot modify it outside the list unless we iterate over the list. Additionally, if we want to update tf.Variable, we should use its assign .etc methods, this is like modifying the content specified by the pointer variable in C language.

More conveniently, we usually use tf.keras.optimizers's apply_gridents(), even minimize() to updata varibales directly.

For this specific task and your more process oriented coding approach, here I give out some suggestions for stable training:

  1. add activations to constrain the fitting ability of this model:
def __call__(self,x):
    Z = x
    num_layers = len(self.layers)
    for i in range(num_layers-2):
        y = tf.math.add(tf.linalg.matmul(Z ,self.weights[i]),self.biases[i])
        Z = tf.nn.relu(y)
    i+=1
    return tf.math.add(tf.linalg.matmul(Z ,self.weights[i]),self.biases[i])
  1. make lower learning_rate:
self.l_r = 0.001 # self.l_r = 0.1
  1. do more epochs:
epochs = range(1000) # epochs = range(10)

Since initial value of trainable wights will also influence the training stability, you may need to re-train several times. In my tests, the above modification works.

user_173579630
Related