I am trying to replicate the model from this paper on page 4 under bullet point 1). To my understanding, it takes a target key (e.g. a place or person), a [SEP] token and then the reply tweet. Since they use a publicly available pretrained bert model, I have tried to use the trainer API from huggingface to replicate it.
However, when executing the trainer I get the error:
TypeError: forward() got an unexpected keyword argument 'label'
I have looked at these issues and questions:
PyTorch BERT TypeError: forward() got an unexpected keyword argument 'labels'
https://github.com/huggingface/transformers/issues/17102
but I think they do not apply, because I DO use the bertforsequenceclassification class.
Since it involves a bit of code, I thought it might be easier to share a google colab link with my code: https://colab.research.google.com/drive/1S3rOUftom_iC7IA_tpK5qeOYXRVRI818
Any help is appreciated. Thanks!
UPDATE:
Seems like the issue was with the tokenization. I had masked a previous error, which stated that sequences were of unexpected length (indicating a wrong padding), by using a data_collator. However, the root problem was the padding that did not work. In the end I had to tokenize using this code:
return tokenizer(examples["text"], padding='max_length', max_length=108)
However, I still do not get the problem. Max_length of 101 would not work, nor did any other value than 108. It would always pad some sequences longer than others. I do not understand why. Maybe a problem with special characters in tweets that would need to be preprocessed? Still looking forward to any input on this.