I would like to save a checkpoint every time a validation loop ends. I set up the val_check_interval to be 0.2 so I have 5 validation loops during each epoch but the checkpoint callback saves the model only at the end of the epoch. I couldn't find an easy (or hard) way to save the model after each validation loop. It seems a bit strange cause I can't see a reason to make the validation loop other then saving a checkpoint.
What is the simplest way of doing that?
Thank you,
Gilad
Have you checked pytorch_lightning.callbacks.model_checkpoint.ModelCheckpoint?
Not sure if it exists on your version but, setting every_n_val_epochs to 1 should work. If so, it should save your model checkpoint after every validation loop.
From the lightning docs: save_on_train_epoch_end (Optional[bool]) – Whether to run checkpointing at the end of the training epoch. If this is False, then the check runs at the end of the validation.
Using save_on_train_epoch_end = False flag in the ModelCheckpoint for callbacks in the trainer should solve this issue.
