How can find similarity between sentences?

user_14383499 0
Viewed 309

I'm trying to find similarities between both the sentences in a shell script.

Have a two sentences containing duplicate words, for example, the input data in file my_text.txt

Shell Script.
Linux Shell Script.

  • The intersection of both sentences: Shell + Script

  • The union " size " of both sentences: 3

The correct output for similarity of sentences :

 0.30000000000000000000

The definition of the similarity ** is the intersection of words between the two sentences divided by the size of the union of the two sentences.

The problem: I have tried a lot to found a shell script, but I have not found a solution to this problem.

awk shell unix linux bash
2 Answers

The following script should do the trick. It also ignores duplicated words per sentence, filler words, and non-alphabetical characters as described by you in the comment section.

words=$(
  < my_text.txt tr 'A-Z' 'a-z' |
  grep -Eon '\b[a-z]*\b' |
  grep -Fwvf <(printf %s\\n is a to be by the and for) |
  sort -u | cut -d: -f2 | sort
)
union=$(uniq <<< "$words" | wc -l)
intersection=$(uniq -d <<< "$words" | wc -l)
echo "similarity is $(bc -l <<< "$intersection/$union")"

The output for your example input is .30000000000000000000 (= 0.3).

user_67703840

Is this what you're trying to do (using GNU awk for FPAT and arrays of arrays)?

$ cat tst.awk
BEGIN {
    split("is a to be by the and for",tmp)
    for (i in tmp) {
        stopwords[tmp[i]]
    }
    FPAT="[[:alnum:]_]+"
}
{
    for (i=1; i<=NF; i++) {
        word = tolower($i)
        if ( !(word in stopwords) ) {
            words[NR][word]
        }
    }
}
END {
    for (word in words[1]) {
        if (word in words[2]) {
            numCommon++
        }
    }
    totWords = length(words[1]) + length(words[2]) - numCommon
    print (totWords ? numCommon / totWords : 0)
}


$ awk -f tst.awk file
0.666667
user_17450010
Related