Which model/technique to use for specific sentence extraction?

user_7605543 0
Viewed 186

I have a dataset of tens of thousands of dialogues / conversations between a customer and customer support. These dialogues, which could be forum posts, or long-winded email conversations, have been hand-annotated to highlight the sentence containing the customers problem. For example:

Dear agent, I am writing to you because I have a very annoying problem with my washing machine. I bought it three weeks ago and was very happy with it. However, this morning the door does not lock properly. Please help

Dear customer.... etc

The highlighted sentence would be:

However, this morning the door does not lock properly.

  1. What approaches can I take to model this, so that in future I can automatically extract the customers problem? The domain of the datasets are broad, but within the hardware space, so it could be appliances, gadgets, machinery etc.
  2. What is this type of problem called? I thought this might be called "intent recognition", but most guides seem to refer to multiclass classification. The sentence either is or isn't the customers problem. I considered analysing each sentence and performing binary classification, but I'd like to explore options that take into account the context of the rest of the conversation if possible.
  3. What resources are available to research how to implement this in Python (using tensorflow or pytorch)

I found a model on HuggingFace which has been pre-trained with customer dialogues, and have read the research paper, so I was considering fine-tuning this as a starting point, but I only have experience with text (multiclass/multilabel) classification when it comes to transformers.

huggingface-transformers pytorch tensorflow nlp python
2 Answers

If you want to get a specific sentence (without any modification) from the original input text, that is often referred to as 'span classification' where the output is the index of the first and last word of the specific sentence. The state-of-the-art now is the attention models like BERT .You can check the Bert models that are designed for the 'span classification' problem in huggingface as RobertaForQuestionAnswering https://huggingface.co/docs/transformers/model_doc/roberta#transformers.TFRobertaForQuestionAnswering that uses TensorFlow or PyTorch library.

user_52355280

This type of problem where you want to extract the customer problem from the original text is called Extractive Summarization and this type of task is solved by Sequence2Sequence models.

The main reason for this type of model being called Sequence2Sequence is because the input and the output of this model would both be text.

I recommend you to use a transformers model called Pegasus which has been pre-trained to predict a masked text, but its main application is to be fine-tuned for text summarization (extractive or abstractive).

This Pegasus model is listed on Transformers library, which provides you with a simple but powerful way of fine-tuning transformers with custom datasets. I think this notebook will be extremely useful as guidance and for understanding how to fine-tune this Pegasus model.

user_170416370
Related