I'm not sure if it's BERT related or not, had no chance to test other models, but did it for BERT.
So what I noticed recently that training algorithms and data that I used to work with in google colab for free, are seemed to work significantly slower in Azure ML workspace which we pay for.
I made the comparison - same data file (classification problem, sentiment analysis of 10K reviews), totally same notebook code (copy+paste), same latest ver of ktrain lib installed on both, both must be on Python 3.8, but GPU is a bit more performant on a colab side.
Results surprised me to say the least: google lab made its job 10 times faster: 17 min vs 170 min, and it's reproducible. Tesla T4 (colab) is faster than K80 (azure) indeed, but not that much as per known benchmarks. So I wonder what else could matter. Is it virt. environment created in Azure ML performing so slow? If you have any idea what it could be, or what else I can check on both sides to reveal it, please share
BTW google gives you T4 in colab for your experimentations for free, while you have to pay for slower K80 at Azure.
Google colab
execution time = 17 min
Google colab hardware: cpu model: Intel(R) Xeon(R) CPU @ 2.20GHz, memory 13Gb, GPU:
Azure
execution time = 2h50m = 170min (10x of colab)
Azure hardware information
K80 and T4 comparison: https://technical.city/en/video/Tesla-K80-vs-Tesla-T4