Python: Get the longest matching keyword mentions in text

user_4551921 0
Viewed 135

I have a list of keywords including their variations that I search in text like :

keywords = ['US Dollar', 'Australian Dollar', 'Dollar', 'Dollars']

and I want to look up these keywords in texts like :

'Dollar News: The Australian Dollar slumped in the face of a recovering US Dollar'

and get the most comprhensive matches (i.e. longest) which are 'Dollar' in the beginning of the sentence, and 'Australian Dollar' and 'US Dollar'(and not 'Dollar' in those cases for instance).

I have so far tried this:

keywords.sort(key = len, reverse=True)

first = lambda text, kws: next((k for k in kws if k in text), None)

first(myText, keywords)

which returns 'Australian Dollar' as it is the longest match. How can I get other matches (here, 'Dollar' in 'Dollar News...' and 'US Dollar') as well?

nlp python
2 Answers
# -*- coding: utf-8 -*-
"""
Created on Thu Jun 13 14:21:59 2019

@author: jainil
"""
keywords = ['US Dollar', 'Australian Dollar', 'Dollar', 'Dollars']
keywords.sort(key = len, reverse=True)
keywords

text='The Australian Dollar slumped in the face of a recovering US Dollar'
dictt={}
for i in keywords:
    dictt[i]=text.count(i)

max_len=0
max_value=0
for i in dictt.keys():
    if len(i.split())>max_len and dictt[i]>0:
        max_len= len(i.split())
        if(dictt[i]>max_value):
            max_value=dictt[i]



for i,j in dictt.items():
    if(len(i.split())==max_len and j==max_value):
        print(i,j)
user_115972150

A solution is to use suffix trees to get the positions of every keyword mention and then handle the overlapping as suggested by @EricDuminil .

Here is my function for extracting keywords' kws positions in text source txt: 

from suffix_trees import STree

def findMentions(txt, kws):

    st = STree.STree(txt)
    spans = []

    for kw in kws:
        starts = st.find_all(kw)
        spans.extend([(item, item+len(kw)) for item in starts])

    bounds = handleOverlap(spans)

    return bounds

and here is the function to handle overlapping character positions:

def handleOverlap(spans):
    del_in = []
    for x in spans:
        if spans.index(x) in del_in: continue
        for y in spans:
            if spans.index(y) in del_in: continue
            if x == y: continue
            if len(set(list(range(x[0],x[1]+1))) & set(list(range(y[0],y[1]+1)))) > 0:
                if len(list(range(x[0],x[1]+1))) > len(list(range(y[0],y[1]+1))):
                    del_in.append(spans.index(y))
                    spans.pop(spans.index(y))
                elif len(list(range(y[0],y[1]+1))) > len(list(range(x[0],x[1]+1))):
                    del_in.append(spans.index(x))
                    spans.pop(spans.index(x))
    return spans

I just had to add spaces to the both ends of each keyword to avoid getting words containing a keyword, like 'petrodollar'. The results are the non-overlapping start and end positions for the longest corresponding mentioned keywords.

user_45519210
Related