avoid infinity for ctc_loss in tensorflow

user_8348248 0
Viewed 980

here is my following code. You can change the activation values but you cannot change the target. The tensorflow ctc_loss always returns Inf. I need ctc_loss to return some floating value and not Inf. I know, the reason for INF (i.e. the activation when multiplied becomes extremely small and hence log of that if Inf). I am interested in a way around it. Change inputs or do whatever except modifying target to get a Non-Inf value.

My code:

inputs = tf.random_uniform([1, 9, 11]) # Do Not Change the Shape of inputs

target = tf.constant([[2,2]], dtype=tf.int32)

zero = tf.constant(0, dtype=tf.float32)
where = tf.ones(tf.shape(target))
indices = tf.where(where)
values = tf.gather_nd(target, indices)

sparse = tf.SparseTensor(indices, values, target.shape)
seq = tf.multiply(tf.ones([tf.shape(target)[0]], dtype=tf.int32), 2)
loss = tf.nn.ctc_loss(sparse, inputs, seq, time_major=False, ctc_merge_repeated=True)

with tf.Session() as sess:
    print (loss.eval())
loss-function tensorflow python-3.x
2 Answers

The problem is the labels you have for the ctc_loss contains repetitions. Setting preprocess_collapse_repeated=True solves it, as its doc describes:

"If preprocess_collapse_repeated is True, then a preprocessing step runs before loss calculation, wherein repeated labels passed to the loss are merged into single labels. This is useful if the training labels come from, e.g., forced alignments and therefore have unnecessary repetitions."

user_54345260

This question is a bit old, but since I had just the same problem, and the current only answer does not seem to solve it (although non-trivially, so much so that in the beginning I believed it was the right answer and upvoted it! But then it failed with other examples...), here is some almost-complete answer.

TL;DR: to make the code in the question work, replace seq = tf.multiply(tf.ones([tf.shape(target)[0]], dtype=tf.int32), 2) by seq = tf.multiply(tf.ones([tf.shape(target)[0]], dtype=tf.int32), 9), i.e. the length of the logits and not of the target.

Details: This does not seem to be properly documented on the manual, and also I am lost as to why it should be so (the shape of the logits can easily be obtained by means of a tf.shape...), but this seems to give the desired result. The reason for this escapes me a bit.

One simple experiment confirms that the logits are actually truncated after sequence_length: running

indices = [[0, 0],
       [0, 1],
       [0, 2]]

values = [0, 1, 1]

shape = [1, 3]

tf.nn.ctc_loss(
    tf.SparseTensor(indices, values, shape),
    [[[ 10., -10., -10.],
        [-10., -10.,  10.],
        [-10.,  10., -10.],
        [-10., -10.,  10.],
        [-10.,  10., -10.]]],
    [5],
    time_major=False,
)

gives almost 0 loss, as it should (replacing the 10s in the logits by bigger numbers one gets closer and closer to 0). Here the argument passed as logits is an approximation of the pre-softmax version of the one-hot encoded vector [0, 2, 1, 2, 1], with 2 being the separating token, therefore correctly decodes to the target "0 1 1".

However, replacing the [5] passed to sequence_length with a [4] still results in a finite loss, but a non-zero one. The reason is that it reads [0, 2, 1, 2], which decodes to the wrong string "0 1". Indeed, switching the rows so that the logits represent [0, 1, 2, 1, 2] or [0, 1, 2, 1, 1] results in the same loss, either passing [4] or [5]. Also, predicting [0, 1, 2, 1, 0] results in 0 loss if passing [4] (since the last 0 is ignored) and, correctly, non-zero loss if passing [5] (since then it decodes to "0 1 0", which is wrong).

All these experiments would be perfect... if they explained why the original example in the question gives infinite. Instead, experimentally it seems that replacing the [2] by [9] in the original question makes it work, but so it does replacing it by [3], [4], etc. The annoying part is that, even with [2], this should not really return inf: it should crop to the first two terms of the logits, and use those to predict something (never inf, since the random logits are always between 0 and 1). The only kind-of-answer I could give myself is that, in fact, sequence_length must always be strictly larger than the length of the target. In this sense, the other suggested answer works: if you pre-process the labels by removing duplicates, the target sequence from "2 2" becomes just "2", hence short enough not to raise this problem. However, setting preprocess_collapse_repeated=True cannot be the right way, because it would impeach to predict strings with repeated characters (e.g. hello world ground truth would become helo world, and logits actually predicting hello world would be penalized)

If somebody has more insight on this, I would appreciate it!

user_43052770
Related