Is there a python function to get "unique" strings in the sense of some similarity measure?

user_7678074 0
Viewed 247

I have a set of strings (in my case it is a column of a pandas dataframe, but it would be ok to consider alternative data structures as list/arrays/...) and I would like to get all "unique" values from that set, where unique is not exact matching but fuzzy matching based on some similarity measure. To give an example, let imagine I have this starting set of strings:

error string
Source and destination checksums do not match 213423 != 647687 transfer-failed
Source and destination checksums do not match 654766 != 987821 transfer-failed
SSL handshake after 1 attempts
SSL handshake after 1 attempts\t
SSL handshake after 1 attempts.\n
Impossible to connect to IP:PORT/PATH{1} User timeout over*
Impossible to connect to IP:PORT/PATH{2} User timeout over*

*where IP, PORT and PATH are placeholders for possibly long strings with completely different characters from option {1} to option {2}.

What I would like as an output is a list of the 3 unique patterns (I marked the third as optional since I guess it would be more tricky):

unique patterns requirement
Source and destination checksums do not match 213423 != 647687 transfer-failed mandatory
SSL handshake after 1 attempts mandatory
Impossible to connect to IP:PORT/PATH{1} User timeout over* optional

I'm aware of some methods for fuzzy matching, for example as in Levenshtein and fuzzywuzzy packages, and I think fuzzywuzzy.fuzz.partial_token_set_ratio and partial_ratio do what I want, but only for comparing 2 strings or one string to all the others (fuzzywuzzy.process.extract), as opposed to all the strings together.

I started implementing myself but I soon realised it is a bit tricky and you need careful considerations in terms of how this scales, so I was wondering whether there's already something available for this purpose. Do you have any suggestions?

Thanks in advance :)

fuzzy-comparison python string
1 Answers

I'm sure there's a better way but using Levenshtein and only being able to compare 2 strings at a time I came up with this:

import Levenshtein as lev


RATIO_LIMIT = 0.7

strings = (
    "Source and destination checksums do not match 213423 != 647687 transfer-failed",
    "Source and destination checksums do not match 654766 != 987821 transfer-failed",
    "SSL handshake after 1 attempts",
    "SSL handshake after 1 attempts\t",
    "SSL handshake after 1 attempts.\n",
    "Impossible to connect to IP:PORT/PATH{1} User timeout over*",
    "Impossible to connect to IP:PORT/PATH{2} User timeout over*",
)

uniques = []

for string in strings:
    if not uniques:
        uniques.append(string)

    for unique in uniques:
        if lev.ratio(unique.lower().strip(), string.lower().strip()) > RATIO_LIMIT:
            break
    else:
        uniques.append(string)

print(uniques)

Now I'm sure you can mess around with the RATIO_LIMIT for better results I just picked a random number for similarity, but does this work with your different values for PATH, IP and PORT because I guess if they're too long this method won't work

user_143162820
Related