I have trained a DistilBERT classification model using huggingface and the the model seems to be working well, with a loss of around 0.3 after testing the best model after training with the code:
trainer.evaluate()
However, upon a new run of trying to load the model and attempting to use the trainer class to evaluate the loss again on the same dataset, the reported loss is horrible and seemingly random, often 7+ but not always the same. This is done largely the same as done for training and uses the same seed:
dataset = DatasetModule(
datasets=config.datasets,
tokenizer=tokenizer_name,
seed=42,)
model = DistilBertForTokenClassification.from_pretrained(
model_path,
num_labels=dataset.label_number,
id2label=dataset.id2tag,
label2id=dataset.tag2id,)
trainer: Trainer = Trainer(
model=model,
train_dataset=dataset.train_dataset,
eval_dataset=dataset.valid_dataset,)
metrics = trainer.evaluate(dataset.valid_dataset)
print(metrics)
At the same time, the performance of model in other metrics, such as f1 after loading the model in the same way are about what I would expect. I suspect and may test using a pickle to save the trainer and see if that gives the expected results, though not ideal.
So my question is why is this happening? Is it a bug? Am I going about the process wrong, and if so how can I go about easily loading a model and testing a dataset on it to get a loss?
Thanks!