I'm trying to train Fast R-CNN model.
for epoch in range(num_epochs):
train_one_epoch(model, optimizer, train_data_loader, device, epoch, print_freq=10)
lr_scheduler.step()
evaluate(model, test_data_loader, device=device)
I got an error message like below:
Epoch: [0] [ 0/44] eta: 0:06:57 lr: 0.000121 loss: 4.3884 (4.3884) loss_classifier: 2.2335 (2.2335) loss_box_reg: 0.5938 (0.5938) loss_objectness: 1.1169 (1.1169) loss_rpn_box_reg: 0.4442 (0.4442) time: 9.4806 data: 0.0647 max mem: 8185
Epoch: [0] [10/44] eta: 0:01:23 lr: 0.001283 loss: 3.0393 (3.2865) loss_classifier: 1.6433 (1.5846) loss_box_reg: 0.6233 (0.6344) loss_objectness: 0.4546 (0.6190) loss_rpn_box_reg: 0.4568 (0.4485) time: 2.4648 data: 0.0562 max mem: 8452
Epoch: [0] [20/44] eta: 0:00:51 lr: 0.002444 loss: 2.3247 (2.7535) loss_classifier: 0.9013 (1.2215) loss_box_reg: 0.6103 (0.6059) loss_objectness: 0.3435 (0.4777) loss_rpn_box_reg: 0.4568 (0.4484) time: 1.7700 data: 0.0529 max mem: 8452
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-26-f72d124a6bf2> in <module>
2 for epoch in range(num_epochs):
----> 3 train_one_epoch(model, optimizer, train_data_loader, device, epoch, print_freq=10)
4 lr_scheduler.step()
...
/usr/local/lib/python3.7/dist-packages/torchvision/models/detection/roi_heads.py in fastrcnn_loss(class_logits, box_regression, labels, regression_targets)
34 # the corresponding ground truth labels, to be used with
35 # advanced indexing
---> 36 sampled_pos_inds_subset = torch.where(labels > 0)[0]
37 labels_pos = labels[sampled_pos_inds_subset]
38 N, num_classes = class_logits.shape
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
After getting this error I cannot even use manuel_seed function.
torch.manual_seed(1)
I got this error again:
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.