How does tensorflow.GradientTape() accumulate operation within a loop?

user_13975623 0
Viewed 409

I'm new to Tensorflow, so this may be a dummy question. I'm trying to figure out what tf.GradientTape() is doing, and I couldn't quite understand how it's recording the operations. Here's the code I tried:

mean = tf.keras.metrics.Mean()
xs = tf.constant([2.0, 3.0, 4.0])
for x in xs:
  with tf.GradientTape() as g:
    y = x * x
  print(y, mean(y))

Abd here's the output:

tf.Tensor(4.0, shape=(), dtype=float32) tf.Tensor(4.0, shape=(), dtype=float32)
tf.Tensor(9.0, shape=(), dtype=float32) tf.Tensor(6.5, shape=(), dtype=float32)
tf.Tensor(16.0, shape=(), dtype=float32) tf.Tensor(9.666667, shape=(), dtype=float32)

It looks like the mean operation gets accumulated across loops, where in loop 1, it computes 4/1 = 4; in loop 2, it computes (4+9)/2=6.5, and in loop 3, it computes (4+9+16)/3 = 9.666667. But what puzzles me is that the square operation (x * x) isn't accumulated, because when y is printed out at each loop, it's just y, and yet when I take the mean of y, it becomes the mean of y + all previous y's.

I can sort of imagine this to be useful when computing loss, as we can easily take the mean of accumulated loss over samples/batches. But i have trouble visualize how tf.GradientTape is "selectively" recording operations behind the scene.

tensorflow python
2 Answers
def get_gradient(x):
   with tf.GradientTape() as tape:
      y = tf.nn.sigmoid(x)
   gradient = tape.gradient(y, x)
   return gradient, y
        
test = tf.Variable([0.5, 0.6, 0.7])
gradient, y = get_gradient(test)

Let's consider a sigmoid activation function. A sigmoid activation function outputs values from 0 to 1.

g(x) = 1/(1 + e^(-x)) = y

g'(x) = g(x)(1 - g(x))

Example: Let, x = 0.5

g(0.5) = 1/(1 + e^(-0.5)) = 0.62245

g'(0.5) = g(x)(1 - g(x)) = 0.2350

########################################################################

with tf.GradientTape() as tape: within this you defined your function you wanted to differentiate. Here, it is y = sigmoid(x). Tape will record all the operations on x.

gradient = tape.gradient(y, x) y = f(x) is a function with y' = f'(x). Input parameters are tensor of y (solved at given x) and x. It returns dy/dx as a tensor.

user_94894650 
def mean(lst):
  return tf.reduce_mean(lst)
  
xs = tf.constant([2.0, 3.0, 4.0])
for x in xs:
  with tf.GradientTape() as g:
    y = x * x
  print(y, mean(y))

output:

tf.Tensor(4.0, shape=(), dtype=float32) tf.Tensor(4.0, shape=(), dtype=float32)
tf.Tensor(9.0, shape=(), dtype=float32) tf.Tensor(9.0, shape=(), dtype=float32)
tf.Tensor(16.0, shape=(), dtype=float32) tf.Tensor(16.0, shape=(), dtype=float32)

As i have used reduce_mean() it only calculates mean of the tensor at that instance but, tf.keras.metrics.mean() accumulates every time its compiled. Hope, this cleared out your doubt.

mean = tf.keras.metrics.Mean()
xs = tf.constant([2.0, 3.0, 4.0])
for i in range(0,2):
  for x in xs:
    with tf.GradientTape() as g:
      y = x * x
    print(y, mean(y))

output:

tf.Tensor(4.0, shape=(), dtype=float32) tf.Tensor(4.0, shape=(), dtype=float32)
tf.Tensor(9.0, shape=(), dtype=float32) tf.Tensor(6.5, shape=(), dtype=float32)
tf.Tensor(16.0, shape=(), dtype=float32) tf.Tensor(9.666667, shape=(), dtype=float32)
tf.Tensor(4.0, shape=(), dtype=float32) tf.Tensor(8.25, shape=(), dtype=float32)
tf.Tensor(9.0, shape=(), dtype=float32) tf.Tensor(8.4, shape=(), dtype=float32)
tf.Tensor(16.0, shape=(), dtype=float32) tf.Tensor(9.666667, shape=(), dtype=float32)
user_94894650
Related