InvalidArgumentError: logits and labels must have the same first dimension seq2seq Tensorflow

user_2359772 0
Viewed 5102

I am getting this error in seq2seq.sequence_loss even though first dim of logits and labels has same dimension, i.e. batchSize

I have created a seq2seq model in TF 1.0 version. My loss function is as follows : 

    logits  = self.decoder_logits_train
    targets = self.decoder_train_targets
    self.loss     = seq2seq.sequence_loss(logits=logits, targets=targets, weights=self.loss_weights)
    self.train_op = tf.train.AdamOptimizer().minimize(self.loss)

I am getting following error on running my network while training : 

InvalidArgumentError (see above for traceback): logits and labels must have the same first dimension, got logits shape [1280,150000] and labels shape [1536]
     [[Node: sequence_loss/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits = SparseSoftmaxCrossEntropyWithLogits[T=DT_FLOAT, Tlabels=DT_INT32, _device="/job:localhost/replica:0/task:0/cpu:0"](sequence_loss/Reshape, sequence_loss/Reshape_1)]]

I confirm the shapes of logits and targets tensors as follows :

a,b = sess.run([model.decoder_logits_train, model.decoder_train_targets], feed_dict)
print(np.shape(a)) # (128, 10, 150000) which is (BatchSize, MaxSeqSize, Vocabsize)
print(np.shape(b)) # (128, 12) which is (BatchSize, Max length of seq including padding)

So, since the first dimension of targets and logits are same then why I am getting this error ?

Interestingly, in error u can observe that the dimension of logits is mentioned as (1280, 150000), which is (128 * 10, 150000) [product of first two dimension, vocab_size], and same for targets i.e. (1536), which is (128*12), again product of first two dimension ?

Note : Tensorflow 1.0 CPU version

tensorflow deep-learning python-3.x
3 Answers

maybe your way of padding wrong. if you padded _EOS to the end of target seq, then the max_length(real length of target sentence) should add 1 to be [batch, max_len+1]. Since you padded _GO and _EOS, your target sentence length should add 2, which makes it equals 12.

I read some other people's implementation of NMT, they only padded _EOS for target sentence, while _GO for input of decoder. Tell me if I'm wrong.

user_97623720

I had the same error as you and I understood the problem:

The problem:

You run the decoder using this parameters:

  • targets are the decoder_inputs. They have length max_length because of padding. Shape: [batch_size, max_length]
  • sequence_length are the non-padded-lengths of all the targets of your current batch. Shape: [batch_size]

Your logits, that are the output tf.contrib.seq2seq.dynamic_decode has shape:

[batch_size, longer_sequence_in_this_batch, n_classes]

Where longer_sequence_in_this_batch is equal to tf.reduce_max(sequence_length)

So, you have a problem when computing the loss because you try to use both:

  • Your logits with 1st dimension shape longer_sequence_in_this_batch
  • Your targets with 1st dimension shape max_length

Note that longer_sequence_in_this_batch <= max_length

How to fix it:

You can simply apply some padding to your logits. 

logits  = self.decoder_logits_train
targets = self.decoder_train_targets

paddings = [[0, 0], [0, max_length-tf.shape(logits)[1]], [0, 0]]
padded_logits = tf.pad(logits, paddings, 'CONSTANT', constant_values=0)


self.loss = seq2seq.sequence_loss(logits=padded_logits, targets=targets, 
                                  weights=self.loss_weights)

Using this method,you ensure that your logits will be padded as the targets and will have dimension [batch_size, max_length, n_classes]

For more information about the pad function, visit Tensorflow's documentation

user_118393450
Related