For a particular task I wish to generate a unique BERT embedding for a list of tokens. Now, BERT by definition does not have a unique representation for every word, but only has a contextual embedding that varies with the context. Given this, what would be the best way to create a mapping from a list of tokens {V} to an embedding space R such that the mapping is one to one?
I am throwing a bunch of ideas out there, please feel free to shoot them down:
Is the average of all the embedding representations of [u] a unique representation? Will this representation satisfy semantic relations between tokens? So will
average_embedding(king) - average_embedding(man) + average_emebedding(woman) = average_embedding(queen)?
The average embedding is generated by summing all the embeddings of [u] in a given corpus and dividing it by the number of occurrences.
- Could a token such as 'bank' be seen a collection of finite embeddings - one say for 'bank_river' and the other for 'bank_money'. So one token [u] can be seen as collection of separate tokens [u1, u2, uk] as long as k is a finite number and the set [u(1 - k)] is such that cosine(uj,u) < e for u in the corpus where e is an arbitrarily small number.
In general, my questions is how to use bert embeddings to create a unique representation for a given word or token.