Embedding in PyTorch creates embedding with norm larger than max_norm

Question

Suppose we have an embedding matrix of 10 vectors with dimension of 100, and we impose max_norm=1:

x = Embedding(num_embeddings=10, embedding_dim=100, max_norm=1)

In principle, every embedding should have norm less or equal to 1. However, when I print the vector norms, I get values much greater than 1:

for w in x.weight: 
    print(torch.norm(w))

> tensor(11.1873, grad_fn=)
> tensor(10.5264, grad_fn=)
> tensor(9.6809, grad_fn=)
> tensor(9.7507, grad_fn=)
> tensor(10.7940, grad_fn=)
> tensor(11.4134, grad_fn=)
> tensor(9.7021, grad_fn=)
> tensor(10.4027, grad_fn=)
> tensor(10.1210, grad_fn=)
> tensor(10.4552, grad_fn=)

Any particular reason why this happens and how to fix it?

user_6316272 · Answer

The max_norm argument bounds the norm of the embedding, but not the norm of the weights.

To better understand this, you can run the following example:

from torch import LongTensor, norm
from torch.nn import Embedding

sentences = LongTensor([[1,2,4,5],[4,3,2,9]])
embedding = Embedding(num_embeddings=10, embedding_dim=100, max_norm=1)
for sentence in embedding(sentences):
    for word in sentence:
        print(norm(word))

This works by dividing each weight in the embedding vector by the norm of the embedding vector itself, and multiplying it by max_norm. In your example max_norm=1, hence it's equivalent to dividing by the norm.

To answer the question you asked in the comment, you can obtain the embedding of a sentence (vector containing word indexes taken from your dictionary), with embedding(sentences), the norm using the 2 for loops above.

Embedding in PyTorch creates embedding with norm larger than max_norm

1 Answers