TensorflowJS while fitting, loss is NaN

Viewed 1415

I am trying to make the same example of the python version of Tensorflow with TensorflowJS. Unfortunately when I run the script, I don't know why the loss value logged while training is NaN.

What I want to achieve is a simple text classification that returns either 0 or 1 based on the trained model. This is the Python tutorial I am following https://www.tensorflow.org/hub/tutorials/text_classification_with_tf_hub

and that's the code I have translated so far:

import * as tf  from '@tensorflow/tfjs'

// Load the binding:
//require('@tensorflow/tfjs-node');  // Use '@tensorflow/tfjs-node-gpu' if running with GPU.

// utils
const tuple = <A, B>(a: A, b: B): [A, B] => [a, b]

// prepare the data, first is result, second is the raw text
const data: [number, string][] = [
    [0, 'aaaaaaaaa'],
    [0, 'aaaa'],
    [1, 'bbbbbbbbb'],
    [1, 'bbbbbb']
]

// normalize the data
const arrayFill = [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]
const normalizeData = data.map(item => {
    return tuple(item[0], item[1].split('').map(c => c.charCodeAt(0)).concat(arrayFill).slice(0, 10))
})

const xs = tf.tensor(normalizeData.map(i => i[1]))
const ys = tf.tensor(normalizeData.map(i => i[0]))

console.log(xs)

// Configs
const LEARNING_RATE = 1e-4

// Train a simple model:
//const optimizer = tf.train.adam(LEARNING_RATE)
const model = tf.sequential();
model.add(tf.layers.embedding({inputDim: 1000, outputDim: 16}))
model.add(tf.layers.globalAveragePooling1d({}))
model.add(tf.layers.dense({units: 16, activation: 'relu'}))
model.add(tf.layers.dense({units: 1, activation: 'sigmoid'}))
model.summary()
model.compile({optimizer: 'adam', loss: 'binaryCrossentropy', metrics: ['accuracy']});

model.fit(xs, ys, {
  epochs: 10,
  validationData: [xs, ys],
  callbacks: {
    onEpochEnd: async (epoch, log) => {
      console.log(`Epoch ${epoch}: loss = ${log.loss}`);
    }
  }
});

(here pure JS code) and that's the output I get

_________________________________________________________________
Layer (type)                 Output shape              Param #
=================================================================
embedding_Embedding1 (Embedd [null,null,16]            16000
_________________________________________________________________
global_average_pooling1d_Glo [null,16]                 0
_________________________________________________________________
dense_Dense1 (Dense)         [null,16]                 272
_________________________________________________________________
dense_Dense2 (Dense)         [null,1]                  17
=================================================================
Total params: 16289
Trainable params: 16289
Non-trainable params: 0
_________________________________________________________________
Epoch 0: loss = NaN
Epoch 1: loss = NaN
Epoch 2: loss = NaN
Epoch 3: loss = NaN
Epoch 4: loss = NaN
Epoch 5: loss = NaN
Epoch 6: loss = NaN
Epoch 7: loss = NaN
Epoch 8: loss = NaN
Epoch 9: loss = NaN
2 Answers

The loss or prediction can become NaN. This is the result of the vanishing gradient problem. During training, the gradient (partial derivative) can become so small (tends to 0). The binarycrossentropy loss function uses the logarithme in the computation. And depending on the mathematical operation involving the logarithme, the result can be NaN.

binary cross entropy

If the weights of the model become NaN, the prediction ŷ is also likely to become NaN and so the loss. The number of epochs can be tuned to avoid this issue. Another way, to solve the issue could be to change the loss or optimizer function.

Having said that, the losses of your code are not NaN. Here is an execution of the code on stackblitz. Also, please note the following answer, where the model is being corrected in order not to predict NaN

Just in case somebody else runs into the same issue: mine was that I forgot to normalize the dataset for training; I am using the Boston Housing problem. See here.

Click on "Show me all the plates, please". and have fun!

Related