Just starting with NLP and following along a course on Udemy. I am trying to compute word similarity using cosine similarity of vectors.
import spacy
nlp = spacy.load('en_core_web_lg')
# Choose the words you wish to compare, and obtain their vectors
word1 = nlp.vocab['wolf'].vector
word2 = nlp.vocab['dog'].vector
word3 = nlp.vocab['cat'].vector
# Import spatial and define a cosine_similarity function
from scipy import spatial
cosine_similarity = lambda x, y: 1 - spatial.distance.cosine(x, y)
# Write an expression for vector arithmetic
# For example: new_vector = word1 - word2 + word3
new_vector = word1 - word2 + word3
# List the top ten closest vectors in the vocabulary to the result of the expression above
computed_similarities = []
for word in nlp.vocab:
if word.has_vector:
if word.is_lower:
if word.is_alpha:
similarity = cosine_similarity(new_vector, word.vector)
computed_similarities.append((word, similarity))
computed_similarities = sorted(computed_similarities, key=lambda item: -item[1])
print([w[0].text for w in computed_similarities[:10]])
['wolf', 'cat', 'i', 'cuz', 'dare', 'u', 'dog', 'she', 'ai', 'ca']
In the above output most of the words are not even close to the new vector. Ideal output according to the tutor should be:
['maned', 'wolfs', 'wolf', 'lynx', 'wolve', 'yotes', 'canids', 'boars', 'foxes', 'wolfdogs']
Another problem is that when I am trying to compute for other words like king, man and queen the output is:
['king', 'woman', 'she', 'who', 'wolf', 'when', 'dare', 'cat', 'was', 'not']
instead of
['king','queen','commoner','highness','prince','sultan','maharajas','princes','kumbia','kings']
In this output words like "wolf" and "cat" are there since I ran the function on the above wolf,cat and dog example first. If I run the king,man and queen example after reloading the model, then "wolf" and "cat" are not there but some other unrelated words.
I have uninstalled and re-installed the model as well as recreated the environment but same result always. Am I doing something wrong? How do I fix this?