How to learn the embeddings in Pytorch and retrieve it later

user_2118596 0
Viewed 8696

I am building a recommendation system where I predict the best item for each user given their purchase history of items. I have userIDs and itemIDs and how much itemID was purchased by userID. I have Millions of users and thousands of products. Not all products are purchased(there are some products that no one has bought them yet). Since the users and items are big I don't want to use one-hot vectors. I am using pytorch and I want to create and train the embeddings so that I can make the predictions for each user-item pair. I followed this tutorial https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html. If it's an accurate assumption that the embedding layer is being trained, then do I retrieve the learned weights through model.parameters() method or should I use the embedding.data.weight option?

pytorch collaborative-filtering embedding
1 Answers

model.parameters() returns all the parameters of your model, including the embeddings.

So all these parameters of your model are handed over to the optimizer (line below) and will be trained later when calling optimizer.step() - so yes your embeddings are trained along with all other parameters of the network.
(you can also freeze certain layers by setting i.e. embedding.weight.requires_grad = False, but this is not the case here).

# summing it up:
# this line specifies which parameters are trained with the optimizer
# model.parameters() just returns all parameters
# embedding class weights are also parameters and will thus be trained
optimizer = optim.SGD(model.parameters(), lr=0.001)

You can see that your embedding weights are also of type Parameter by doing so:

import torch
embedding_maxtrix = torch.nn.Embedding(10, 10)
print(type(embedding_maxtrix.weight))

This will output the type of the weights, which is Parameter:

<class 'torch.nn.parameter.Parameter'>

I'm not entirely sure what mean by retrieve. Do you mean getting a single vector, or do you want just the whole matrix to save it, or do something else?

embedding_maxtrix = torch.nn.Embedding(5, 5)
# this will get you a single embedding vector
print('Getting a single vector:\n', embedding_maxtrix(torch.LongTensor([0])))
# of course you can do the same for a seqeunce
print('Getting vectors for a sequence:\n', embedding_maxtrix(torch.LongTensor([1, 2, 3])))
# this will give the the whole embedding matrix
print('Getting weights:\n', embedding_maxtrix.weight.data)

Output:

Getting a single vector:
 tensor([[-0.0144, -0.6245,  1.3611, -1.0753,  0.5020]], grad_fn=<EmbeddingBackward>)
Getting vectors for a sequence:
 tensor([[ 0.9277, -0.1879, -1.4999,  0.2895,  0.8367],
        [-0.1167, -2.2139,  1.6918, -0.3483,  0.3508],
        [ 2.3763, -1.3408, -0.9531,  2.2081, -1.5502]],
       grad_fn=<EmbeddingBackward>)
Getting weights:
 tensor([[-0.0144, -0.6245,  1.3611, -1.0753,  0.5020],
        [ 0.9277, -0.1879, -1.4999,  0.2895,  0.8367],
        [-0.1167, -2.2139,  1.6918, -0.3483,  0.3508],
        [ 2.3763, -1.3408, -0.9531,  2.2081, -1.5502],
        [-0.5829, -0.1918, -0.8079,  0.6922, -0.2627]])

I hope this answers your question, you can also take a look at the documentation, there you can find some useful examples as well.

https://pytorch.org/docs/stable/nn.html#torch.nn.Embedding

user_74834940
Related