How to get accuracy during/after training for Huggingface RobertaForMaskedLM model?

user_9495110 0
Viewed 599

I am using HuggingFace Trainer to train a Roberta Masked LM. I am passing the following function for compute_metrics as other discussion threads suggest:

metric = load_metric("accuracy")

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)

I am loading my dataset from text files using load_dataset, and after applying a tokenizer, it returns attention_id, input_ids but no labels. I am training on this dataset using Trainer. Still, I cannot find the accuracy of my model during training after passing the compute_metrics function above or cannot evaluate my model after training on test data. How to get accuracy for this model during training and evaluate it as we could do in Keras models using model.evaluate()? How is it measured?

huggingface-transformers pytorch keras python
1 Answers

If you are using one of the example scripts, have you checked the --evaluation_strategy parameter? The default value is None, but it can be set to steps or epoch.

user_124287820
Related