Understanding the parameters of a simple neural network in Pytorch

user_10796158 0
Viewed 94

I have defined a simple NN as follows in Pytorch: 

my_model = nn.Sequential(
            nn.Linear(1, 5),
            nn.Tanh(),
            nn.Linear(5, 1))

I then iterate through the parameters and check their sizes:

[parameter.shape for parameter in my_model.parameters()]

I get:

[torch.Size([5, 1]), torch.Size([5]), torch.Size([1, 5]), torch.Size([1])]

I'm confused as to why the last Size is 1. Shouldn't there be 5 bias values as well, going out from the hidden layer into the output layer?

pytorch python
2 Answers

Um, so it looks like you are defining a two layers network.

The input size is [1], and the hidden layer size is [5], and next, the hidden layer is connected to the output layer, the output layer size is [1].

It should look like:

       *   <- input
   / / | \ \
   * * * * *  <- hidden layer
   \ \ | / /
       *   <- output

So for the hidden layer, there are 5 values as bias, for the output layer, there is only one value as bias. Sounds reasonable?

user_55882790

The number of biases in any layer depends on the number of output nodes of that layer. Since your final layer has one output node, it only has one bias. It applies to Convolutional layers as well, in that case, number biases is number of output channels you define.

user_118631950
Related