Does doc2vec model give accuracy on non-dictionary words?

user_15036998 0
Viewed 49

I have sentences in corpus with mixed words (dictionary and non-dictionary words). Non-dictionary words are as important as they are domain specific. I'm not performing any nlp on non-dictionary words. Does doc2vec model compare non-dictionary words to same words in matching criteria?

Ex. I'm giving input ['AMDML','release']. Here AMDML is domain specific word. Will it match to same words if I've sentences in training model like ['AMDML','release','process'] or ['DML','release']. or only words like 'release' and 'process' is matched in most similar method?

doc2vec gensim python
1 Answers

I guess not;

According to radimrehurek-gensim page which mentioned Le and Mikolov paper (introducers of Doc2Vec algorithm), They refer to the Paragraph Vector model as Doc2Vec;

In Gensim, we refer to the Paragraph Vector model as Doc2Vec. Which usually outperforms such simple-averaging of Word2Vec vectors. The basic idea is: act as if a document has another floating word-like vector, which contributes to all training predictions, and is updated like other word-vectors, but we will call it a doc-vector. Gensim’s Doc2Vec class implements this algorithm.

So i guess Doc2Vec just follow Word2Vec model/algorithm; As i know if Word2Vec model for example has AMDML word on its training corpus it can generate a vector for it; Otherwise it has know idea about that and show something like error: missing word to you or at least returns padding/empty vector.

I think you need something like fasttext; fasttext model always has vector for any words even if they don't exist on its traianing corpus; Unlike word2vec, fasttext can learn from n-gram character of words so you can find similar words by measuring their similarity values. After that for each sentence/doc averaging these similarities and find similar sentences/docs.

user_73434040
Related