Training a BERT model and using the BERT embeddings

user_13514161 0
Viewed 1434

I've been reading up on BERT and using BERT embeddings for a classification task. I've read many articles but my understanding of it still is not 100% (I have self-taught myself NLP, so my access to resources can be a bit restricted). First I'll describe my task.

I was planning on using BERT embeddings for classification because of how it encapsulates the meaning and language. Unfortunately, there are no BERT models in my language (Irish), so I looked into training my own. I know that BERT is basically an 'extension' of sorts to a Transformer Encoder.

Here are my issues/questions:

  • I presume this is fairly obvious, but to check, pre-trained BERT embeddings cannot be applied to different languages (the standard embedding model is trained on the wiki dataset for English, I presume it may not be used on other languages for obvious reasons)?

  • My datset contains about 850k sentences in Irish (around 22M words). Would that be enough to train a decent BERT model? I could find a bit more data but to get significantly more in Irish would be very hard.

  • Would one recommend to make a BERT model 'from scratch' in PyTorch or TensorFlow, or are models from the likes of Fairseq and OpenNMT good to use?

Apologies for such a disjointed question, but in summary, I'm all over the place trying to make complete sense of BERT, specifically the training process and tuning it just for embeddings. If I've got this all wrong, or just have advice, I'd appreciate the feedback.

bert-language-model pytorch tensorflow nlp python
2 Answers

I'm just like you, a self knowledge teacher of NLP. Since you have not started yet (you'll have such a quiet journey, don't you?), I would recommend you to check this proyect in tensorflow library since it's from Google and you'll have better access to all its power (just my opinion):

First, you'll need a vocab file to tokenize with: it's a file (txt) which contains a fixed size of strings, one per line. BERT uses around 30.000, so think about your number aswell (higher number doesn't mean higher accuracy). This tutorial of tokenization will help you

If you have a deep curiosity about how transformer works, then check out this one in addition

In terms of training from scratch a new BERT model, take a look at this question: Can you train a BERT model from scratch with task specific architecture?

You'll need a very powerfull computer to be able to do this. If not, you'll have a lot of memory issues. In the other hand, Tensorflow allows you to train its hub models (both preprocessing and encoder) so I think there is no need to reinvent the wheel. For this, use tensorflow_hub (also install tensorflow_text since I think you'll have dependency errors). I let you here the link for tfdev hub of each preprocessing and encoder models of BERT (if you download one, in asset folder you'll find the vocab file).

850K sentences and 22M words may are less data than used in the original BERT. If your proyect it's just about curiosity, then it's big enough.

Hope I've helped you! Good luck

PD: I'm starting to use BERT aswell. Your question is from August last year so I think you have been able to make some progress. I'll appreciate and be interested in read it!

user_127128480

While not in the title of the question, your question seems to be how can you obtain a BERT model for Irish. There are now three monolingual Irish BERT models available:

  • Since May 2020, there is wikibert-ga trained on Irish Wikipedia (~ 0.7 million sentences).
  • Since August 2020, there is BERTreach. The model card says it is trained on 47 million tokens (2.1 million sentences), including the PARSEME verbal multiword expressions corpus for Irish, presumably using the unannotated raw portion. Furthermore they use Corpus Crawler and a small corpus, presumably from https://wortschatz.uni-leipzig.de/en/download/Irish (the link in the model card does not work).
  • Since July 2021, there is gaBERT trained on 7.9 million sentences, including not freely available corpora. (There is some overlap between data sources. Only counting the clearly disjoint sources, there are at least 4.7 million sentences.)

Furthermore, the multilingual LaBSE model may be of interest to you as it is specifically trained to encode the meaning of sentences.

Disclaimer: I am co-author of the gaBERT paper.

user_50231290
Related