What is the initial value of Embedding layer?

user_3548063 0
Viewed 1405

I am studying embedding for word representations. In many dnn libraries, they support embedding layer. And this is really nice tutorial.

Word Embeddings: Encoding Lexical Semantics

But I am not still sure how to calculate embed value. In below example, it outputs some value even before any trainings. Does it use some random weights? I realize a purpose of Embedding(2, 5), but not sure its initial calculation. And I am no sure about how to learn weights of its Embedding too.

word_to_ix = {"hello": 0, "world": 1}
embeds = nn.Embedding(2, 5)  # 2 words in vocab, 5 dimensional embeddings
lookup_tensor = torch.LongTensor([word_to_ix["hello"]])
hello_embed = embeds(autograd.Variable(lookup_tensor))
print(hello_embed)
--------
Variable containing:
-2.9718  1.7070 -0.4305 -2.2820  0.5237
[torch.FloatTensor of size 1x5]

I break down my thought to be sure. First of all, upper Embedding(2, 5) is a matrix of shape (2, 5).

Embedding(2, 5) = 
 [[0.1,-0.2,0.3,0.4,0.1],
 [-0.2,0.1,0.8,0.2,0.3]] # initiated by some function, like random normal distribution

Then, hello is [1, 0]. Then hello representation is calculated by [1, 0].dot(Embedding(2, 5)) = [0.1,-0.2,0.3,0.4,0.1]. This is actually first row of the Embedding. Am I understanding right?

Updates

I found a code of embedding which is exactly use normal distribution for its value. Yes, but it is just a default value, and we can set arbitrary weights for embedding layers. https://github.com/chainer/chainer/blob/adba7b846d018b9dc7d19d52147ef53f5e555dc8/chainer/links/connection/embed_id.py#L58

word2vec word-embedding deep-learning machine-learning nlp
2 Answers

Initializations define the way to set the initial random weights of layers. You can use any value to do it. But initial values affect Word Embedding. There are many approach for Pre-trained Word Embedding that they try to choose better initial values like this.

user_61344080

Yes. You start off with random weights. I think it is more common to use a truncated normal distribution instead of the regular normal distribution. But, that probably doesn't make much of a difference.

user_28780040
Related