How to group similar sentences using sklearn consine similarities

user_11131258 0
Viewed 85

I have a pandas series description which I calculated the similarities between the sentences using sklearn

0            0114600043776001 loan payment receipt
1                   ogsg s u b e b june 2018 salar
2                                 sal admin charge
3                     sms alert charge outstanding
4     vat onverve*issuance fee*506108*********1112
5           verve*issuance fee*506108*********1112
6              visa credit card repayment jul 2018
7          trsf 0043776013 12140114fcmb ijebu ode1
8                      maint fee recovery jun 2018
9                  vat maint fee recovery jun 2018
10           0114600043776001 loan payment receipt
11           0114600043776001 loan payment receipt
12                             ogsg subeb july sal
13                    sms alert charge outstanding
14         trsf 0043776013 12141363fcmb ijebu ode2
15                     maint fee recovery jul 2018
16                 vat maint fee recovery jul 2018
17         recry card maintenance charge july 2018
18                              ogsg subeb aug sal
19              433090995 wd 10322883 15 ibadan rd



def cosine_sim(description):
    vectorizer = TfidfVectorizer(min_df=1,analyzer='word', ngram_range=(1, 3), stop_words='english')
    tfidf_matrix = vectorizer.fit_transform(description)
    # similarities of this doc
    matches = cosine_similarity(tfidf_matrix, tfidf_matrix)

    return matches

The cosine_sim function returns an array of matrix with values between(0,1). Now I want to matches sentences whose similarity is between the range 0.2 & 0.99999

similarities = cosine_sim(description)
nums = similarities[(0.2<similarities) & (similarities<=0.99999)] and return a list of list, I simple can't figure out a way around this.

my expected output should look like this

['0114600043776001 loan payment receipt',
  '0421209017073500 loan payment receipt'],
 ['ogsg s u b e b june 2018 salar'],
 ['sal admin charge'],
 ['sms alert charge outstanding'],
 ['vat onverve*issuance fee*506108*********1112'],
 ['verve*issuance fee*506108*********1112'],
 ['visa credit card repayment jul 2018',
  'visa credit card repayment sep 2018',
  'visa credit card repayment oct 2018',
  'visa credit card repayment nov 2018',
  'visa credit card repayment aug 2018'],
 ['trsf 0043776013 12140114fcmb ijebu ode1',
  'trsf 0043776013 12141363fcmb ijebu ode2'],
 ['maint fee recovery jun 2018',
  'vat maint fee recovery jun 2018',
  'maint fee recovery jul 2018',
  'vat maint fee recovery jul 2018',
  'maint fee recovery aug 2018',
  'vat maint fee recovery aug 2018',
  'maint fee recovery oct 2018',
  'vat maint fee recovery oct 2018',
  'maint fee recovery nov 2018',
  'vat maint fee recovery nov 2018',
  'maint fee recovery may 2018',
  'vat maint fee recovery may 2018',
  'maint fee 29 jun 2018 30 jul 2018',
  'vat maint fee 29 jun 2018 30 jul 2018',
  'maint fee 31 jul 2018 30 aug 2018']]
scikit-learn pandas scipy python
1 Answers

First, we need to remeber that cosine_similarity(tfidf_matrix, tfidf_matrix) return a matrix of similarity using tfidf_matrix indexes, therefore to this case matches[i][j]==matches[j][i] and that value represent the similarity between description[i] and description[j].

We can get the index of each match of each description with this:

matchs_sentences={}
#for each description
for index in similarities:
    #get match index for that description 
    matchs_sentences[index]= similarities[(0.2<similarities[index]) & (similarities[index]<=0.99999)].index

We will create a "match group" for each description:

matchs_text=[]
#for each "match group"
for index in matchs_sentences :
    # get group sentenses
    m =list( description[matchs_sentences[index]])
    # insert self on result
    m.append(description[index])

The final value of matchs_text is like that:

[['0114600043776001 loan payment receipt'],
 ['ogsg s u b e b june 2018 salar'],
 ['sal admin charge'],
 ['sms alert charge outstanding'],
 ['verve*issuance fee*506108*********1112',
  'vat onverve*issuance fee*506108*********1112'],
 ['vat onverve*issuance fee*506108*********1112',
  'verve*issuance fee*506108*********1112'],
 ['visa credit card repayment jul 2018'],
 ['trsf 0043776013 12141363fcmb ijebu ode2',
  'trsf 0043776013 12140114fcmb ijebu ode1'],
 ... ]

! Note that have intersections and duplicate strings. If you dont want intersections you use something like this:

matchs_text_without_intersections=[]
matched=[]
for index in matchs_sentences :
    #for each description not matched in a group
    if index not in matched:
        #get match index for that description 
        index_matches =matchs_sentences[index]
        matched.append(index)
        group=[description[index]]

        for index2 in index_matches:
            #for each index2 that match index and are not in a group
            if index2 not in matched:
                group.append(description[index2])
                matched.append(index2)
        matchs_text_without_intersections.append(group)
user_98980330
Related