NLP Paragraph Level Predictions vs Doc Level Predictions? What Strategy to deploy

user_10391992 0
Viewed 33

currently wanted to understand which model an approach I am incorporating for model development, I currently have a TF-IDF NLP model that reads in paragraphs for a document and makes a prediction based upon how many paragraphs scored a 1 label with that paragraph.

I am not sure if that is correct form of logic, should I just go with an document level model? what are the benefits and trade-offs of predicting at a paragraph level and rolling it up into a total prediction for the document vs just classifying the document itself.

Any Thoughts?

Thanks!

word2vec doc2vec tf-idf machine-learning nlp
1 Answers

Depends on what problem you are trying to solve and the nature of your data.

If in one document different parts can be classified differently, it's better to make a prediction by paragraphs or even sentences. For example - quite often, the customer can be happy with one part of the product/item (the first sentence is positive). And be dissatisfied with another part of the product/item (the second sentence has a negative sentiment).

Or, if the document is entirely related to a specific topic, you can make a prediction using the entire text.

In the end, these are just assumptions. Hold out a test subset and validate your model for both cases.

user_86093300
Related