Update of the weights after Gradient Descent in TensorFlow

user_1795641 0
Viewed 1536

I am new to tensorflow and the neural net. I am trying to understand that, how the weights are updated after the Gradient Descent function executed? The example code is as below.

with graph.as_default():

    weights = tf.Variable(
    tf.truncated_normal([image_size * image_size, num_labels]))
    biases = tf.Variable(tf.zeros([num_labels]))

    logits = tf.matmul(train_dataset, weights) + biases
    loss = tf.reduce_mean(
    tf.nn.softmax_cross_entropy_with_logits(labels=train_labels, logits=logits))
    loss=loss+tf.multiply(beta, nn.l2_loss(weights))

    optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss) 

with tf.Session(graph=graph) as session:
    tf.global_variables_initializer().run()
    _, l, predictions = session.run([optimizer, loss, train_prediction])

If I understand correctly, when running the “session.run()” the variables weights and the biases will be updated. Will it be updated in the context of whatever values “GradientDescentOptimizer” has count or it will be just another set of the “truncated_normal” values?

If the regularization is applied as below,

loss=loss+tf.multiply(beta, nn.l2_loss(weights))

Then, how the tensorflow will know what is the correct variable to update the weights in the context of regularized weights? I am not getting the working of the TF.

tensorflow neural-network python
2 Answers

Take a look at following picture from Tensorflow official website that explains about Graph and Session concepts:

enter image description here

According to documentation:

  • Calling tf.constant() creates a single Operation that produces a value, adds it to the default graph.
  • Calling tf.matmul(x, y) creates a single Operation that multiplies the values of tf.Tensor objects x and y, adds it to the default graph, and returns a tf.Tensor that represents the result of the multiplication
  • Calling tf.train.Optimizer.minimize will add operations and tensors to the default graph that calculates gradients, and return a Operation that, when run, will apply those gradients to a set of variables.

when running the “session.run()” the variables weights and the biases will be updated.

Actually their value calculated not updated. For example, take a look at following example:

a = tf.Variable(2)
with tf.Session() as sess:
    sess.run(a.initializer)
    print(sess.run(a))

In this example no update will happen.

Look at the above picture again, as you can see in the picture when we go forward we understand what parameters needs to updated so in the backward, the parameters updated according to loss by SGD optimizer.

user_14627700

Initially weights and biases are initialised using random values. When you run session.run([...]), it'll evaluate optimizer,loss and train_prediction and all the variables these three may depend on.

For example, optimizer depends on loss, loss on train_labels and logits, logits on weights and biases and so on...

When it reaches the end(calculates all variables), it'll update the weights and biases as per gradient descent algorithm(To understand, how tensorflow does it, you'll need to understand Gradient Descent algorithm first. Check out this link). It's called "completing 1 epoch". In your case, you have used only 1 epoch so there will be only one pass. Accuracy won't be that good either. To further optimize it, use it like below:

Let epochs=100

with tf.Session(graph=graph) as session::
     tf.global_variables_initializer().run()
     for i in range(epochs):
         _, l, predictions = session.run([optimizer, loss, train_prediction])

This way, session.run(...) will run 100 times, updating the weights and biases in every iteration according to loss.

Tensorflow will update all those varibles, which are initialised using tf.Variable().

user_98652250
Related