Steps for Machine Learning in Pytorch

user_4442478 0
Viewed 38

When we define our model in PyTorch. We run through different #epochs. I want to know that in the iteration of epochs. What is the difference between the two following snippets of code in which the order is different? These two snippet versions are:

  1. I found over tutorials
  2. The code provided by my supervisor for the project.

Tutorial Version

for i in range(epochs):
    logits = model(x)    
    loss = loss_fcn(logits,lables)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

Supervisor Version

for i in range(epochs):
    logits = model(x)
    loss = loss_fcn(logits,lables)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
pytorch machine-learning python
3 Answers

The only difference is when the gradients are cleared. (when you call optimizer.zero_grad()) the first version zeros out the gradients after updating the weights (optimizer.step()), the second one zeroes out the gradient after updating the weights. both versions should run fine. The only difference would be the first iteration, where the second snippet is better as it makes sure the residue gradients are zero before calculating the gradients. Check this link that explains why you would zero the gradients

user_165025570

In PyTorch, we typically want to explicitly set the gradients to zero for every mini-batch during the training phase before starting backpropagation (i.e., updating the Weights and biases) because PyTorch accumulates the gradients on subsequent backward passes. Regarding your question, both snippets do the same, the important detail is calling optimizer.zero_grad() before loss.backward().

user_60574830

Here is a pseudo code for the iteration:

  1. run model
  2. compute loss

<-- zero grads here...

  1. go backward (compute grads if no grads otherwise accumulate)
  2. update weights

<-- ...or here

Basically you zero grads before or after going backward and updating the weights. Both code snippets are OK.

user_107494320
Related