I trained a RoBERTa model following this colab - https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb#scrollTo=XaFAsB_fnU3K
Here is how my data looked:
Merkel bemoans lack of rain as Germany fears for its forests .\n
Germany’s forests, covering a third of its territory and as much a part of its cultural landscape as its physical one, are in danger.\n
An aerial view shows a forest near Gummersbach, Germany, April 24, 2020, following an unusually warm, dry winter after a summer of record temperatures leaving forests dried out.\n
Picture taken with a drone.\n
The last two exceptionally hot and dry summers have weakened millions of trees, undermining their defences against the bark beetle, which can be fatal to ancient woodlands.\n
And after an exceptionally dry April, with summer still two months away, a forest fire has already had to be put out near the town of Gummersbach in western Germany this week.\n
“We’re already noticing these days that it’s not raining enough in many areas.
After training the model I used pipeline from the transforms library for the fill_mask task
from transformers import pipeline
fill_mask = pipeline(
"fill-mask",
model="./output",
tokenizer="./output"
fill_mask("Merkel bemoans lack of rain as <mask> fears for its forests")
)
These are the results:
[{'sequence': '<s> Merkel bemoans lack of rain as. fears for its forests</s>',
'score': 0.040456026792526245,
'token': 18},
{'sequence': '<s> Merkel bemoans lack of rain as, fears for its forests</s>',
'score': 0.03502459451556206,
'token': 16},
{'sequence': '<s> Merkel bemoans lack of rain as the fears for its forests</s>',
'score': 0.03497963398694992,
'token': 269},
{'sequence': '<s> Merkel bemoans lack of rain as\n fears for its forests</s>',
'score': 0.03180328756570816,
'token': 203},
{'sequence': '<s> Merkel bemoans lack of rain as to fears for its forests</s>',
'score': 0.020796578377485275,
'token': 288}]
As you can see there is no meaningful word(s) returned only punctuations and one other word (to) which doesn't make sense. What am i doing wrong here? Do I have to remove all punctuations?