Fine-tune BERT for a specific domain on a different language?

user_7988187 0
Viewed 928

I want to fine-tune on a pre-trained BERT model. However, my task uses data within a specific domain (say biomedical data). Additionally, my data is also in a language different from English (say Dutch).

Now I could fine-tune the Dutch bert-base-dutch-cased pre-trained model. However, how would I go about fine-tuning a Biomedical BERT model, like BioBERT, which is in the correct domain, but wrong language?

I have thought about using NMT, but don't think it's viable and worth the effort. If I fine-tune without any alterations to the model, I fear that the model will not learn the task well since it was pre-trained on a completely different language.

bert-language-model deep-learning nlp python-3.x
1 Answers

I just want to know if there are any methods that allow for fine-tuning a pre-trained BERT model trained on a specific domain and use it for data within that same domain, but a different language

Probably not. BERT's vocabulary is fixed at the start of pre-training, and adding additional vocabulary leads to random weight initializations.

Instead, I would:

  1. Look for a multi-lingual, domain-specific version of BERT as @Ashwin said.
  2. Fine-tune Dutch BERT on your task and see if performance is acceptable. In general, BERT can adapt to different tasks quite well.
  3. (If you have the available resources) Continue pre-training Dutch BERT on your specific domain (for example, like SciBERT) and then fine-tune on your task.
user_143909300
Related