I am trying to understand 'epochs' in neural network training. Are the next experiments equivalent?

user_8499604 0
Viewed 210

Lets say I have a training sample (with their corresponding training labels) for a defined neural network (the architecture of the neural network does not matter for answering this question). Lets call the neural network 'model'.

In order to not create any missunderstandings, lets say that I introduce the initial weights and biases for 'model'.

Experiment 1.

I use the training sample and the training labels to train the 'model' for 40 epochs. After the training, the neural network will have a specific set of weights and biases for the entire neural network, lets call it WB_Final_experiment1.

Experiment 2

I use the training sample and the training labels to train 'model' for 20 epochs. After the training, the neural network will have a specific set of weights and biases for the entire neural network, lets call it WB_Intermediate.

Now I introduce WB_Intermediate in 'model' and train for another 20 epochs. After the training, the neural network will have a specific set of weights and biases for the entire neural network, lets call it WB__Final_experiment2.

Considerations. Every single parameter, hyperparameter, activation functions, loss functions....is exactly the same for both experiments, except the epochs.

Question: Are WB_Final_experiment1 and WB__Final_experiment2 exactly the same?

keras machine-learning epoch neural-network python
3 Answers

If the operations you are doing are entirely deterministic, then yes. Epochs are implemented as an iteration number for a for loop around your training algorithm. You can see this in implementations in PyTorch.

user_113249940

Typically no, the model weights will not be the same as the optimisation will accrue its own values during training. You will need to save those too to truly resume from where you left off. See the Pytorch documentation regarding saving and resuming here. But this concept is not limited to the Pytorch framework.

Specifically:

It is important to also save the optimizer’s state_dict, as this contains buffers and parameters that are updated as the model trains.

user_101887370

If you follow this tutorial here, you will find the results of the two experiments as given below -

Experiment 1

enter image description here

enter image description here

Experiment 2

enter image description here

enter image description here

In the first experiment the model ran for 4 epochs and in the second experiment, the model ran for 2 epochs and then trained for 2 more epochs using last weights of previous training. You will find that the results vary but to a very small amount. And they will always vary due to the randomized initialization of weights. But the prediction of both models will lie very near to each other.

If the models are initialized with the same weights then the results at the end of 4 epochs for both the models will remain same.

On the other hand if you trained for 2 epochs, then shut down your training session and the weights are not saved and if you train now for 2 epochs after restarting session, the prediction won't be the same. To avoid that before training, always load the saved weights to continue training using model.load_weights("path to model").

TL;DR

If models are initialized with the exact same weights, then the output at the end of same training epochs will remain same. If they are randomly initialized the output will only vary slightly.

user_123334550
Related