Does it differ to use optimizer.step or model.step in pytorch?

user_5223033 0
Viewed 4095

In pytorch, to update the model, should I use optimizer.step() or model.step() ?

Here is a example snippet:

import torch
import torch nn
class SomeNeuralNet(nn.Module):
    def __init__(self,hs,es,dropout):
          SomeNeuralNet(ClaimRecognizer, self).__init__()
          # Some initialization here
    def forward(x):
          # forward propagation here

model = SomeNeuralNet(es,hs,dp)
optimizer = optim.Adam(model.parameters())
loss_function = nn.NLLLoss()
for epoch in N:
   for x in data:
      logp = model(x)
      loss = loss_function(logp,gold_outs)
      loss.backward()
       # Which one I should call ? Optimizer.step() or model.step() or both ?
       optimizer.step()
       model.step()
pytorch python
2 Answers

To make the gradient descent step, you normally use just optimizer.step().
Here is also an example taken from the documentation (same link at bottom), what it looks like in general:

for input, target in dataset:
    optimizer.zero_grad()
    output = model(input)
    loss = loss_fn(output, target)
    loss.backward()
    optimizer.step()

I don't know where you got this model.step()? Does did you try it?

If your model really possesses some kind of step()-functionality, it probably does something different.

But unless you define something extra, your model gets its functions from nn.Module and this does not have step function!

See this example from the the Pytorch Documentation:

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

model = Model()
model.step()

Trying to call step() result in an AttributeError:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-41-b032813f7eda> in <module>
     13 
     14 model = Model()
---> 15 model.step()

~/miniconda3/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    530                 return modules[name]
    531         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 532             type(self).__name__, name))
    533 
    534     def __setattr__(self, name, value):

AttributeError: 'Model' object has no attribute 'step'

To sum it up, normally your model should not have a step() function, optimizer.step() is the way to go if you want to do the optimization step.

See also here: https://pytorch.org/docs/stable/optim.html#taking-an-optimization-step

user_74834940

There no model.step() in pytorch, you may check your code again. I guess you created step function when you were creating Model. If so, you may use model.Module.step() to realize step function.

user_172391420
Related