When do you want to explicitly call model.forward(input) in PyTorch?

user_5194677 0
Viewed 244

In most PyTorch examples, I see out = model(input) instead of out = model.forward(input). I understand that the latter doesn't handle any hooks, and the first option is generally preferred.

However, in the Annotated Transformer implementation of the Attention Is All You Need Transformer, they use model.forward() when getting the output from the entire model, but all layers within the model are called using layer(x) instead of layer.forward(x). I assume this explicit use of .forward() is intentional, but can't figure out why.

Edit: Very sorry for not being clear on what I meant by layer. The code I am referring to has structure like:

class model(nn.Module):

    def __init__(self, layer1, layer2, layer3):
        super(model, self).__init__()
        self.layer1 = layer1
        self.layer2 = layer2 
        self.layer3 = layer3

    def forward(self, x):
        return self.layer2(self.layer1(x))

class layer1(nn.Module):

    def __init__(self, inputs):
        super(layer3, self).__init__()
        ## set stuff here

    def forward(self, x):
        ## do stuff to x
        return x

class layer2(nn.Module):

    def __init__(self, layer, N):
        super(layer2, self).__init__()
        ## set stuff here

    def forward(self, x):
        ## do more stuff to x
        return x

As you can see, within model, x is propagated through all the layers by just layer(x), but in this particular example, the outputs are calculated by out = model.forward(input).

pytorch class oop python
1 Answers

When you define your model, you have something that probably looks like this:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        
        self.layer_1 = nn.Linear(10, 64)
        self.layer_2 = nn.Linear(64, 64)
        self.layer_out = nn.Linear(64, 1)
        
        self.relu = nn.ReLU()
        self.dropout = nn.Dropout(p = 0.1)
        self.batchnorm1 = nn.BatchNorm1d(64)
        self.batchnorm2 = nn.BatchNorm1d(64)
        
    def forward(self, inputs):
        x = self.relu(self.layer_1(inputs))
        x = self.batchnorm1(x)
        x = self.relu(self.layer_2(x))
        x = self.batchnorm2(x)
        x = self.dropout(x)
        x = self.layer_out(x)
        
        return (x)

forward() is a method of your model object, not your layer object. A layer object can take input as an argument, but you cannot call forward() on a layer because there is no forward method for these objects.

Hopefully this makes sense.

user_137463940
Related