Is there a way to perform concept matching of text in Python?

user_18086289 0
Viewed 116

I am trying to write a Python program which takes in multiple strings of text along with a phrase, and assigns each text string a score based on how much the concept discussed in the phrase actually appears in the text.

I want this to be a little more sophisticated than just a synonym finder, perhaps more similar to the kinds of searches Google, Google Scholar, or Semantic Scholar perform.

My current algorithm just tokenizes the phrase and runs individual synonym searches on each word, and pieces all the different combinations together to form new phrases. The results are pretty mediocre.

As an example, if I had the phrase "user-centered approach," I would like to be able to flag text strings that discussed things like human-computer interaction or human-focused design, even if the exact phrase is never used in the text.

Is there a way to achieve this or somethings similar in Python?

topic-modeling text-processing nlp python
1 Answers

If you're not striving for the last percent fractions of SOTA improvements in text classification and just seeking an affordable yet efficient solution I recommend using zero shot learning. Huggingface transformers make it super easy, you need to provide the text and list of phrases!

from transformers import pipeline
classifier = pipeline("zero-shot-classification")
sequences = [
             "is a framework of process in which usability goals, user characteristics, environment, tasks and workflow of a product, service or process are given extensive attention at each stage of the design process.",
             "the science and art of preventing disease, prolonging life and promoting health through the organized efforts and informed choices of society, organizations, public and private, communities and individuals"
            ]

candidate_labels = ["politics", "public health", "user-centered approach"]

classifier(sequences, candidate_labels)

output:

[{'labels': ['user-centered approach', 'public health', 'politics'],
  'scores': [0.9876492023468018, 0.007956252433359623, 0.00439451914280653],
  'sequence': 'is a framework of process in which usability goals, user characteristics, environment, tasks and workflow of a product, service or process are given extensive attention at each stage of the design process.'},
 {'labels': ['public health', 'user-centered approach', 'politics'],
  'scores': [0.6508855819702148, 0.31257015466690063, 0.03654429316520691],
  'sequence': 'the science and art of preventing disease, prolonging life and promoting health through the organized efforts and informed choices of society, organizations, public and private, communities and individuals'}]
user_87041800
Related