Loss returns nan in tensorflow

user_16605625 0
Viewed 200

I am training an autoencoder whose input is a matrix P in [0,1] using the following loss function:

enter image description here

And here is my code:

# Define loss and optimizer, minimize the squared error
with tf.device("/device:GPU:0"):
     L = -tf.reduce_sum(self.p*tf.math.log(self.p_pred+1e-10) + 0.55*(1 - self.p)*tf.math.log(1-self.p_pred+1e-10), axis = 1)
     self.loss = tf.reduce_mean(L)
self.optimizer = tf.compat.v1.train.AdamOptimizer(self.learning_rate).minimize(self.loss)

In main.py, I run session like this:

_, l, y_pred, y = sess.run([model.optimizer, model.loss, model.y_pred, model.y], feed_dict=...)

But the loss returns nan at different epoch whenever I'm training. The activation function is sigmoid and learning_rate = 0.01. The number of epoch is 20. I'm trying to save p and p_pred when the loss is nan, then I run the same loss function in google colab, the result is not nan! I don't understand.

Any ideas for what I did wrong?

loss-function tensorflow nan python
1 Answers

The loss function you're using is BCELoss which gives "NaN" when the value inside log is less than or equal to 0. Sometimes, your term inside the log might go <=0 and sometimes, it might not. That's why you might not be getting NaN in colab.

Try adding sigmoid in the loss function (like sigmoid(self.p_pred) instead of self.p_pred. This will ensure the term is in (0,1) range.

Else, try increasing the epsilon value (1e-10 in your code. Increase that value to 1e-5 may be)

user_132849440
Related