TF gradient returning zero for simple problem

Viewed 915

In TF 2.0, I traced back a bug and isolated to this simple problem: tf.gradient is giving zero gradient, it should be infinity.

bxe = tf.keras.losses.BinaryCrossentropy(); 
a=tf.Variable(1.0) 
with tf.GradientTape() as tape: 
   loss = bxe([0.], a) 
grads = tape.gradient(loss, a) # compute gradients 
print(loss, grads) 

tf.Tensor(15.333239, shape=(), dtype=float32) 
tf.Tensor(0.0, shape=(), dtype=float32)

As you can see, the loss should be infinite cause -log(1-1) in the binary cross entropy. However, I looked up documentation and I noticed that they clip the numbers before applying the log, that's why 15.333 show up instead of infinity. That's great, and help avoiding many troubles during training, BUT, what about the gradient? It is producing zero! According to my math it should be infinite, so, at least it should be producing something very high, rather than zero. As such, my training is stuck! Why is that happening? How did I even arrive at this point? and how come that this doesn't happen when I use Keras high level built classifier models?

0 Answers
Related