RuntimeError: "exp" not implemented for 'torch.LongTensor'

user_5833067 0
Viewed 10882

I am following this tutorial: http://nlp.seas.harvard.edu/2018/04/03/attention.html to implement the Transformer model from the "Attention Is All You Need" paper.

However I am getting the following error : RuntimeError: "exp" not implemented for 'torch.LongTensor'

This is the line, in the PositionalEnconding class, that is causing the error:

div_term = torch.exp(torch.arange(0, d_model, 2) * -(math.log(10000.0) / d_model))

When it is being constructed here: 

pe = PositionalEncoding(20, 0)

Any ideas?? I've already tried converting this to perhaps a Tensor Float type, but this has not worked.

I've even downloaded the whole notebook with accompanying files and the error seems to persist in the original tutorial.

Any ideas what may be causing this error?

Thanks!

attention-model tensor pytorch
5 Answers

I happened to follow this tutorial too.

For me I just got the torch.arange to generate float type tensor

from 

position = torch.arange(0, max_len).unsqueeze(1)
div_term = torch.exp(torch.arange(0, d_model, 2) * -(math.log(10000.0) / d_model))

to

position = torch.arange(0., max_len).unsqueeze(1)
div_term = torch.exp(torch.arange(0., d_model, 2) * -(math.log(10000.0) / d_model))

Just a simple fix. But now it works for me. It is possible that the torch exp and sin previously support LongTensor but not anymore (not very sure about it).

user_84986860

It seems like torch.arange returns a LongTensor, try torch.arange(0.0, d_model, 2) to force torch to return a FloatTensor instead.

user_17144100

The suggestion given by @shai worked for me. I modified the init method of PositionalEncoding by using 0.0 in two spots:

position = torch.arange(0.0, max_len).unsqueeze(1)
div_term = torch.exp(torch.arange(0.0, d_model, 2) * -(math.log(10000.0) / d_model))
user_108684750

For me, installing pytorch == 1.7.1 solved the problem.

user_69016900

Like Rubens said, in the higher version of Pytorch, you don't need to worry about this stuff. I can easily run it on my desktop's 1.8.0 Pytorch, but failed to go through it in my server's 1.2.0 Pytorch. There is something incompatible between different versions.

user_123043670
Related