There is a part that I cannot understand when I train the deep learning model on colab.
When I first train the model, measuring training loss and valid loss at the same time, it works for every batch and epoch.
However, when I train again in the same cell, or train another model in a different cell, RuntimeError: cuda runtime error (59) : device-side assert triggered error happens suddenly.
I also have tried this one at the very front of Colab window, but it didn't work
import os
os.environ['CUDA_LAUNCH_BLOCKING'] = "1"
if os.environ.get('COLAB_GPU', '0') == '1':
os.environ['GPU_NUM_DEVICES'] = '1'
os.environ['XLA_FLAGS'] = '--xla_gpu_cuda_data_dir=/usr/local/cuda/'
I don't think it is because of index error from loss function, because if it was an index error, the first trial should not work! Here is be the link where I got information, but not being my case.
Is here anybody who can solve this problem?