Fine-tune a BERT model for context specific embeddigns

Question

I'm trying to find information on how to train a BERT model, possibly from the Huggingface Transformers library, so that the embedding it outputs are more closely related to the context o the text I'm using.

However, all the examples that I'm able to find, are about fine-tuning the model for another task, such as classification.

Would anyone happen to have an example of a BERT fine-tuning model for masked tokens or next sentence prediction, that outputs another raw BERT model that is fine-tuned to the context?

Thanks!

user_8979274 · Answer

Here is an example from the Transformers library on Fine tuning a language model for masked token prediction.

The model that is used is one of the BERTForLM familly. The idea is to create a dataset using the TextDataset that tokenizes and breaks the text into chunks. Then use a DataCollatorForLanguageModeling to randomly mask tokens in the chunks when traing, and pass the model, the data and the collator to the Trainer to train and evaluate the results.

Fine-tune a BERT model for context specific embeddigns

1 Answers