Transformer model fine-tuning on single GPU

user_16706366 0
Viewed 474

Is fine-tuning a pre-trained transformer model a easier model an ‘easier’ task than training a transformer from scratch (BERT, GPT-2) in terms of GPU needs and GPU memory usage?

To clarify further, I’ve read how to train most transformer models, one would require multi-GPU training. However, is it possible to fine-tune some of these models on a single-GPU? Why is this the case?

Is it because we can use with smaller batches, the fine-tuning time is not as much as training from scratch?

fine-tune tensorflow2.0 huggingface-transformers machine-learning
1 Answers

Yes, fine-tuning a pre-trained Transformer model is a typical way to go. The training hours required are prohibitively large (even hundred of thousands of hours of decent GPU card per model) while fine-tuning can be done on a single GPU. The reason is that fine-tuning entails training only a few layers on top of the output of the pre-trained model to tailor for a given task. As such, fine-tuning requires less data and significantly less training time to achieve good results.

user_65739020
Related