Print texts that have cosine similarity score less than 0.90

user_9749124 0
Viewed 133

I want to create deduplication process on my database. I want to measure cosine similarity scores with Pythons Sklearn lib. between new texts and texts that are already in the database.

I want to add only documents that have cosine similarity score less than 0.90. This is my code:

from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
from sklearn.metrics.pairwise import cosine_similarity


list_of_texts_in_database = ["More now on the UK prime minister’s plan to impose sanctions against Russia, after it sent troops into eastern Ukraine.",
                             "UK ministers say sanctions could target companies and individuals linked to the Russian government.",
                             "Boris Johnson also says the UK could limit Russian firms ability to raise capital on London's markets.",
                             "He has suggested Western allies are looking at stopping Russian companies trading in pounds and dollars.",
                             "Other measures Western nations could impose include restricting exports to Russia, or excluding it from the Swift financial messaging service.",
                             "The rebels and Ukrainian military have been locked for years in a bitter stalemate, along a frontline called the line of control",
                             "A big question in the coming days, is going to be whether Russia also recognises as independent some of the Donetsk and Luhansk regions that are still under Ukrainian government control",
                             "That could lead to a major escalation in conflict."]


list_of_new_texts = ["This is a totaly new document that needs to be added into the database one way or another.",
                     "Boris Johnson also says the UK could limit Russian firm ability to raise capital on London's market.",
                     "Other measure Western nation can impose include restricting export to Russia, or excluding from the Swift financial messaging services.",
                     "UK minister say sanctions could target companies and individuals linked to the Russian government.",
                     "That could lead to a major escalation in conflict."]


vectorizer = TfidfVectorizer(lowercase=True, analyzer='word', stop_words = None, ngram_range=(1, 1))


list_of_texts_in_database_tfidf = vectorizer.fit_transform(list_of_texts_in_database)
list_of_new_texts_tfidf = vectorizer.transform(list_of_new_texts)

cosineSimilarities = cosine_similarity(list_of_new_texts_tfidf, list_of_texts_in_database_tfidf)
print(cosineSimilarities)

This code works good, but I do not know how to map the results (how to get texts that have similarity score less than 0.90)

scikit-learn python
2 Answers

My suggestion would be as follows. You only add those texts with a score less than (or equal) 0.9.

import numpy as np

idx = np.where((cosineSimilarities <= 0.9).all(axis=1))

Then you have the indices of the new texts in list_of_new_texts that do not have a corresponding text with a score of > 0.9 in the already existing list list_of_texts_in_database.

Combining them you can do as follows (although somebody else might have a cleaner method for this...)

print(
    list_of_texts_in_database + list(np.array(list_of_new_texts)[idx[0]])
)

Output:

['More now on the UK prime minister’s plan to impose sanctions against Russia, after it sent troops into eastern Ukraine.',
 'UK ministers say sanctions could target companies and individuals linked to the Russian government.',
 "Boris Johnson also says the UK could limit Russian firms ability to raise capital on London's markets.",
 'He has suggested Western allies are looking at stopping Russian companies trading in pounds and dollars.',
 'Other measures Western nations could impose include restricting exports to Russia, or excluding it from the Swift financial messaging service.',
 'The rebels and Ukrainian military have been locked for years in a bitter stalemate, along a frontline called the line of control',
 'A big question in the coming days, is going to be whether Russia also recognises as independent some of the Donetsk and Luhansk regions that are still under Ukrainian government control',
 'That could lead to a major escalation in conflict.',
 'This is a totaly new document that needs to be added into the database one way or another.',
 'Other measure Western nation can impose include restricting export to Russia, or excluding from the Swift financial messaging services.',
 'UK minister say sanctions could target companies and individuals linked to the Russian government.']
user_86018070

why dont you work within a dataframe?

import pandas as pd

d = {'old_text':list_of_texts_in_database[:5], 'new_text':list_of_new_texts, 'old_emb': list_of_texts_in_database_tfidf[:5], 'new_emb': list_of_new_texts_tfidf}

df = pd.DataFrame(data=d)

df['score'] = df.apply(lambda row: cosine_similarity(row['old_emb'], row['new_emb'])[0][0], axis=1)
df = df.loc[df.score > 0.9, 'score']
df.head()
user_134739140
Related