PyTorch Average Accuracy after each epoch

Question

I am trying to calculate the accuracy of the model after the end of each epoch. After each epoch I would like to calculate the accuracy over the previous epoch. The model only seems to print the same value as the mean test error.

model.eval()
for images, paths in tqdm(loader_test):
    images = images.to(device)
    targets = torch.tensor([metadata['count'][os.path.split(path)[-1]] for path in paths]) # B
    targets = targets.float().to(device)

    # forward pass:
    output = model(images) # B x 1 x 9 x 9 (analogous to a heatmap)
    preds = output.sum(dim=[1,2,3]) # predicted cell counts (vector of length B)

    # logging:
    loss = torch.mean((preds - targets)**2)
    count_error = torch.abs(preds - targets).mean()
    mean_test_error += count_error
    writer.add_scalar('test_loss', loss.item(), global_step=global_step)
    writer.add_scalar('test_count_error', count_error.item(), global_step=global_step)
    
    global_step += 1

average_accuracy = 0 
mean_test_error = mean_test_error / len(loader_test)
writer.add_scalar('mean_test_error', mean_test_error.item(), global_step=global_step)
average_accuracy += mean_test_error
average_accuracy = average_accuracy /len(loader_test)
print("Average accuracy: %f" % average_accuracy)
print("Test count error: %f" % mean_test_error)
if mean_test_error < best_test_error:
    best_test_error = mean_test_error
    torch.save({'state_dict':model.state_dict(),
                'optimizer_state_dict':optimizer.state_dict(),
                'globalStep':global_step,
                'train_paths':dataset_train.files,
                'test_paths':dataset_test.files},checkpoint_path)

user_4614404 · Answer

IF your model is a classifier, calculating accuracy follows:

acc = (pred.max(dim=1) == target).float().mean()

Where:

pred.shape =  (batch_size, n_classes)
target.shape = (batch_size)

A similar question was asked recently: calculate accuracy for each class using CNN and pytorch

However, you are using MSE loss to train your network. Which is a regression loss. For classification problems CrossEntropyLoss is the GO TO option: https://pytorch.org/docs/master/generated/torch.nn.CrossEntropyLoss.html

IF you are solving an regression task. Than usually accuracy is defined by a threshold. Predictions that are bellow a certain error from the ground truth are considered, the others arent. In this case the calculation is the following:

errors = (pred - target) ** 2  # Squared error
acc = (errors < threshold).float().mean()
error = errors.mean()

Where:

pred.shape = (batch_size, chanels, ...) 
target.shap = (batch_size, chanels, ...)

Was it something like that you were looking for?

PyTorch Average Accuracy after each epoch

1 Answers