How to fuzzy match sentence and keywords in a dataframe?

user_17051008 0
Viewed 101

I tried to see if the sentence contains any words in the keywords (fuzzy match). It works well in the single line but when I run through my dataframe, it returns []

def word2vec(word):
    from collections import Counter
    from math import sqrt

    # count the characters in word
    cw = Counter(word)
    # precomputes a set of the different characters
    sw = set(cw)
    # precomputes the "length" of the word vector
    lw = sqrt(sum(c*c for c in cw.values()))

    # return a tuple
    return cw, sw, lw

def cosdis(v1, v2):
    # which characters are common to the two words?
    common = v1[1].intersection(v2[1])
    # by definition of cosine distance we have
    return sum(v1[0][ch]*v2[0][ch] for ch in common)/v1[2]/v2[2]

list_of_keywords = ['Clearing', 'grubbing']
Sentence = 'Clear and grub Low density Light vegetation'
def list1(Sentence, list_of_keywords):
    a = [x for x in Sentence.str.split() for y in list_of_keywords if cosdis(word2vec(x), word2vec(y)) > 0.7]
    return a

This is a code I used for function:

df_fill = df_fill.astype('str')
df_fill['lol']=df_fill.apply(lambda x: list1(x['Line Item Description'],x['Keyword']), axis=1)

Is there any faster way to do it as my dataframe has 10000 rows? For this input:

list_of_keywords = ['Clearing', 'grubbing']
Sentence = 'Clear and grub Low density Light vegetation'

The expected output is:

['Clear','grub']

Added in a new column.

Example dataframe:

Sentence                                    | keyword             | Result
Clear and grub Low density Light vegetation | ['Clearing', 'grubbing'] | ['Clear','grub']
pandas data-cleaning python
1 Answers

Here is one simple way to do it using Python standard library difflib module, which provides helpers for computing deltas.

from difflib import SequenceMatcher

# Define a simple helper function
def ratio(a, b):
    return SequenceMatcher(None, a, b).ratio()

And with the following toy dataframe:

df = pd.DataFrame(
    {
        "sentence": [
            "Clear and grub Low density Light vegetation",
            "Can you imagine what our life would be like without electricity",
            "Tom pretended to not hear a thing",
        ],
        "keywords": [
            ["Clearing", "grubbing"],
            ["light", "frame"],
            ["pretend", "Hearing"],
        ],
    }
)

You can apply it like this:

df = df.assign(
    result=df.apply(
        lambda x: [
            word
            for keyword in x["keywords"]
            for word in x["sentence"].split(" ")
            if ratio(keyword, word) > 0.5
        ],
        axis=1,
    )
)

Which gives you the expected result:

print(df)
# Output
                                            sentence              keywords             result
0        Clear and grub Low density Light vegetation  [Clearing, grubbing]      [Clear, grub]
1  Can you imagine what our life would be like wi...        [light, frame]                 []
2                  Tom pretended to not hear a thing    [pretend, Hearing]  [pretended, hear]
user_112460560
Related