I am training my model using PyTorch, and I have already added saving and resume training functionalities, but the problem is that when I want to override the step value on tensorboard, the graph gets missed up (connecting the last step to the current step)
Example:
I am logging scalars to tensorboard every 10 iterations, and saving models every 100 iterations where iteration=step.
In the image below, I trained the model for 150 iterations (so I now have 15 step points in the tensorboard graph), and then I resume training from another 100 iterations and start logging in step 100, I was expected to override all step points after step 110 but it seems to connect the last step point (150) to step 110, how I can override all steps after step 100, and fix the graph ???
logging to tensorboard code:
if (OLD_ITERATIONS + i)%10 is 0 and i is not 0:
wirter.add_scalar(tag='Accuracy/Train', scalar_value=accs/10, global_step=OLD_ITERATIONS+i)
wirter.add_scalar(tag='Accuracy/Valid', scalar_value=val_accs/10, global_step=OLD_ITERATIONS+i)
wirter.add_scalar(tag='Loss/Train', scalar_value=losses/10, global_step=OLD_ITERATIONS+i)
wirter.add_scalar(tag='Loss/Valid', scalar_value=val_losses/10, global_step=OLD_ITERATIONS+i)
accs = 0
losses = 0
val_accs = 0
val_losses = 0
