NLP and spaCy: How to find a similar phrase in a string

user_10605692 0
Viewed 176

thanks in advance for reading.

I'm in Python and using spaCy for processing English text. I have a phrase that I want to search for

search_phrase = "payment date"

in a larger phrase

text_to_be_searched = "Party A will pay Party B on the transaction date."

And I want the search to match "payment date" with "transaction date" based on similarity.

How can I do this? I don't see an obvious way, and the only thing I can think of is to manually split text_to_be_searched into chunks. One extra difficulty here is that the matching phrases could have different numbers of tokens, so I'd have to break it into chunks of 1, 2, ... 5 tokens and search each chunk in each set of chunks. For clarity, this would be:

Set of 1-token chunks:

['Party','A','will', ..., 'date']

Set of 2-token chunks:

['Party A','A will','will pay', ..., 'transaction date']

etc

spacy nlp python
1 Answers

You can get the word embeddings then find a similarity between them. spacy is able to do this as follows. According to the documentation, by default, it is cosine similarity. You need to find an optimum threshold value of course.

nlp = spacy.load("en_core_web_md")
search = nlp("payment date")
text_to_be_searched = nlp("Party A will pay Party B on the transaction date.")
threshold = 0.8

matched_words = []
for token in text_to_be_searched:
    print(token, token.similarity(search))
    if token.similarity(search) > threshold:
        matched_words.append(token)

print(f"\nMatched words: {matched_words}")

This prints

Party 0.34560599567510353
A 0.2068122164970917
will 0.7228409255656658
pay 0.7228409255656658
Party 0.34560599567510353
B 0.11308183304731666
on 0.20214880588221648
the 0.324707772449963
transaction 0.9999999409847675
date 0.9999999409847675
. 0.2766332837661776

Matched words: [transaction, date]

To use the en_core_web_md, first you need to download it as follows

python3 -m spacy download en_core_web_md

To get the word embeddings, you can use different things such as language models like BERT etc.

user_131470680
Related