finding semantic similarity between 2 statements

Viewed 624

I am currently working with small application in python and my application has search functionality (currently using difflib) but I want to create Semantic Search which can give top 5 or 10 results from my database, based on user inputted text. It is same as google search engine works. I found some solutions Here.

But the problem is, below two statements from one of solution are semantically incorrect. And I don't care about this. because they are making things too hard which I don't want And also solution will be some pretrained neural network model or library from which I can implement easily.

  • Pete and Rob have found a dog near the station.
  • Pete and Rob have never found a dog near the station

And also I found some solutions which are showing using gensim and Glove embeddings and finding similarity between words and not sentences.

What I wanted ?

Suppose my db has statement display classes and user inputs show, showed, displayed, displayed class, show types etc are same. And if above 2 statements are given as same then also I don't care. displayed and displayed class already showing in difflib.

Points to be noted

  • Find from fixed set of statements but user inputted statements can differ
  • Must work for statements
3 Answers

I think it is not gensim embedding. It is word2vec embedding. Whatever it is.

You need tensorflow_hub

The Universal Sentence Encoder encodes text into high-dimensional vectors that can be used for text classification, semantic similarity, clustering and other natural language tasks.

I believe you need here is Text Classification or Semantic Similarity because you want to find nearest top 5 or 10 statements given statement from user.

It is easy to use. But size of model is ≈ 1GB. It works with words, sentences, phrases or short paragraphs. The input is variable length English text and the output is a 512 dimensional vector. You can find more information about it Here

Code

import tensorflow_hub as hub
import numpy as np

# Load model. It will download first time.
module_url = "https://tfhub.dev/google/universal-sentence-encoder-large/5" 
model = hub.load(module_url)

# first data[0] is your actual value
data = ["display classes", "show", "showed" ,"displayed class", "show types"]

# find high-dimensional vectors.
vecs = model(data)

# find distance between statements using inner product
dists = np.inner(vecs[0], vecs)

# print dists
print(dists)

Output

array([0.9999999 , 0.5633253 , 0.46475542, 0.85303843, 0.61701006],dtype=float32)

Conclusion

First value 0.999999 is distance between display classes and display classes itself. second 0.5633253 is distance between display classes and show and last 0.61701006 is distance between display classes and show types.

Using this, you can find distance between given input and statements in db. then rank them according to distance.

You can use wordnet for finding synonyms and then use these synonyms for finding similar statements.

import nltk
from nltk.corpus import wordnet as wn

nltk.download('wordnet')

def get_syn_list(gword):
  syn_list = []
  try:
    syn_list.extend(wn.synsets(gword,pos=wn.NOUN))
    syn_list.extend(wn.synsets(gword,pos=wn.VERB))
    syn_list.extend(wn.synsets(gword,pos=wn.ADJ))
    syn_list.extend(wn.synsets(gword,pos=wn.ADV))
  except :
    print("Something Wrong Happened")
  syn_words = []
  for i in syn_list:
    syn_words.append(i.lemmas()[0].name())
  return syn_words

Now use split and split your statements in db. like this

stat = ["display classes"]

syn_dict = {}
for i in stat:
   tmp = []
   for x in i.split(" "):
       tmp.extend(get_syn_list(x))
   syn_dict[i] = set(tmp)

Now you have synonyms just compare them with inputted text. And use lemmatizer before comparing words so that displayed become display.

Related