I'm learning nlp, and as a study project, i'm trying to face the news summarization dataset, using BERT.
The dataset is simple (in the news_summary_more.csv) - it has articles and headlines columns.
What I think, is to use the articles as feature (X), and headlines as target (Y).
My question is how do I measure success in the fine-tuning, so the model will learn?
The model prediction will almost never be the same the the real target (y != ^y), because there are so many options to summarize the articles, and almost 100% of the cases the model will have somewhat different summarization than the real headline.
Example:
Team A won the game against team B
is different from:
Team A is victorious against team B on the game
Although both has pretty much the same meaning.
So how can I test if the model is getting close to the real target? (maybe somehow using MSE)