How can I use SpaCy to compute the average similarity between every string item of a list?

user_15952594 0
Viewed 158

I am trying to ascertain how semantically inter-related the items within a given list are. For example, for the following list of nouns, I would like to have the average similarity score across every possible pair (without doing so manually pair by pair, as written also below).

wordlist=['molds', 'brick', 'pipe', 'hand', 'iron']

doc1 = nlp("molds")
doc2 = nlp("brick")

Similarity of two documents
print(doc1, "<->", doc2, doc1.similarity(doc2))

I assume that there is some way to do this through loops, however I am still a beginner to Python so I am struggling to think what the syntax would be. Any suggestions would be greatly appreciated!

Thanks,

Juan

spacy nlp python
1 Answers
wordlist=['molds', 'brick', 'pipe', 'hand', 'iron']
docs = [nlp(word) for word in wordlist]

sims = []
for left in docs: # for every doc
    for right in docs[1:]: # for every doc except the first
        sims.append(left.similarity(right))


import statistics
print(statistics.mean(sims))

Couple of things:

  1. This is slow because you are computing many pairwise similarities. If you need big lists you can compare distance to the average of the embeddings of the list.
  2. spaCy is intended for use on documents (complete sentences or paragraphs), not isolated words. If you just want to use vectors for individual words maybe look at gensim.
user_3557150
Related