Tensorflow multiple session.run( ) in same iteration

user_10056036 0
Viewed 1784

Following are 2 code patterns for the training part of Tensorflow NN.

I find it logical to use the model 1. But I am seeing model 2 in multiple places frequently. I feel Model 2 is wrong. Wont model run the graph in session twice for every iteration for the same data? Is there something I am missing and people do it for any other reasons?

Model 1

for epoch in range(epochs):
    for iteration in range(num_tr_iter):
        _, loss, accuracy = sess.run([optimizer, loss, accuracy], feed_dict)

Model 2

for epoch in range(epochs):
    for iteration in range(num_tr_iter):
        sess.run(optimizer, feed_dict)
        loss, accuracy = sess.run([loss, accuracy],feed_dict)

EDIT : Am expanding the question for more clarity

If the below sess.run() execute optimizer node, it will execute all its dependent node. It will run the underlying convnet and loss function as well.

    sess.run(optimizer, feed_dict)

Next if the below sess.run() execute the loss node, why doesnt it execute the convnet with its current weights. I am not inferring that it will run the optimization again. Even to arrive at the current loss, wouldnt tensorflow execute the convnet and calculate the loss?

        loss, accuracy = sess.run([loss, accuracy],feed_dict)
tensorflow neural-network python
2 Answers

Session.Run will only run calculate the graph element(s) given to in the first argument.

So in Model 2, sess.run(optimizer, feed_dict) just applies the weight updates for the model, and loss, accuracy = sess.run([loss, accuracy],feed_dict) just calculates the loss and accuracy on the model after weights have been updated. The whole computation isn't being run twice since these operations are independent of each other.

Note that the Session.Run docs mention that:

Order in which... operations are evaluated inside the call is undefined.

So Model 1 might not even give an accurate loss and accuracy, since it's unclear if it was measured before or after the weight update. (In practice this might be fine, though)

user_81637910

In model 2, the first line of code runs the optimizer on the model, and the second line calculates the loss and accuracy. It is not running the graph twice.

Model 1 and Model 2 are essentially the same, but points go to model 1 for brevity.

user_103957010
Related