How to find most similar string values in a dataframe?

user_11376275 0
Viewed 33

I am finding the similarity between the sentence using embedding sentence and looping through all the document's embedded sentences to find the right match relative to the search string. I also want to display the document name in the output along with the similarity match result but am not sure how I can extract that information from the dataframe respective to the sentence we get in the output result. I have tried the index method but it is not showing me the correct document name.

Please guide how can I get the document name in the result output along with the sentence.

My data frame looks like this:

Document name        Document sentences in tokens

Doc 1                 [Sentence 1, sentence 2, sentence 3]

Doc 2                 [Sentence 1, sentence 2, sentence 3]

I have used the following code to find the top 10 matches with the search string.

from itertools import chain
docs_sent_tokens=list(chain.from_iterable(main_df['Sentence_Tokenize_rules']))
docs_name=main_df['Document name']


results=[]

#set the threshold value to get the similarity result accordingly
threshold=0

#embedding all the documents and find the similarity between search text and all the tokenize sentences
for docs_sent_token in docs_sent_tokens:

#To find the document name
    for index in main_df.index:
        doc_name= main_df['Document name'][index]

    sentence_embeddings = model.encode(docs_sent_token)
    sim_score1 = cosine_sim(search_sentence_embeddings, sentence_embeddings)
    if sim_score1 > threshold:
            results.append((
                docs_sent_token,
                sim_score1,
                doc_name

                ))

#printing the top 10 matching result in dataframe format
df=pd.DataFrame(results, columns=['Matching Sentence','Similarity Score','Docuemnt name'])

# sorting in descending order based on the similarity score
df.sort_values("Similarity Score", ascending = False, inplace = True)

#change the value of n to see more results
df.head(n=10)

Output should be like this:

Matching sentence    similarity score    document name
Sentence 12              0.80            doc 1
sentence 15              0.69            doc 3
pandas nlp search
1 Answers

Here is an example of how you can do it using Python standard library difflib module, which provides helpers for computing deltas.

Given the following toy dataframe and search sentence:

import pandas as pd

df = pd.DataFrame(
    {
        "document": ["doc 1", "doc 2"],
        "sentences": [
            ["lore ipsum", "magna carta", "upside down"],
            ["tempus fugit", "memento mori", "lora ipsom"],
        ],
    }
)
search_sentence = "lor ipsum"

Define a helper function to compare sentences similarity:

def ratio(a, b):
    return round(SequenceMatcher(None, a, b).ratio(), 2)

And then:

# Use Python instead of Pandas
df = df.to_dict(orient="list")

# Init empty dictionary
results = {"Matching sentence": [], "similarity score": [], "document name": []}

# Iterate to compare
for (doc, sentences) in zip(df["document"], df["sentences"]):
    for i, sentence in enumerate(sentences):
        results["Matching sentence"].append(f"Sentence {i+1}")
        results["similarity score"].append(ratio(search_sentence, sentence))
        results["document name"].append(doc)

Finally:

new_df = (
    pd.DataFrame(results)
    .sort_values(by="similarity score", ascending=False)
    .reset_index(drop=True)
)

print(new_df)
# Ouptut
  Matching sentence  similarity score document name
0        Sentence 1              0.95         doc 1
1        Sentence 3              0.84         doc 2
2        Sentence 2              0.29         doc 2
3        Sentence 3              0.20         doc 1
4        Sentence 1              0.19         doc 2
5        Sentence 2              0.10         doc 1
user_112460560
Related