Neural network : Unexpected matrix size at the output of each layer

user_471461 0
Viewed 78

I am trying to learn Neural networks and the math associated on my own.

I tried to code a simple NN and compare it with my hand calculations, but there's a discrepency that I'm not able to understand and I hope that you can help me.

  1. I wrote the following NN (from this tutorial)
def sigmoid(z):
    return 1.0/(1.0+np.exp(-z))

class Network:
    # sizes is a list of the number of nodes in each layer
    def __init__(self, sizes, biases, weights):
        self.num_layers = len(sizes)
        self.sizes = sizes
        self.biases = biases
        self.weights = weights

    def feedforward(self, a):
        for b, w in zip(self.biases, self.weights):
            a = sigmoid(np.dot(w, a) + b)
        return a
  1. I initialized a 3 layer NN : 2-input, 3-hidden, 1-output
sizes = [2,3,1]
biases = [np.random.randn(y, 1) for y in sizes[1:]]
weights = [np.random.randn(y, x) for x,y in zip(sizes[:-1], sizes[1:])]
net = Network(sizes, biases, weights)
  1. After calculating by hand the outputs of each layer, I compared it with my program by debugging the a variable in the feedforward function

I found these 2 discrepencies that I can't explain

  1. In my program, the output of the hidden layer is a 3x3 matrix. While I expected a 1x3 matrix (each of the 3 hidden neurons outputs a single value). My values matched the diagonal of that 3x3 matrix, but I don't understand what the other values mean
  2. The output of the last layer is a 1x3 matrix, While I expected a 1x1 matrix (one output neuron, one value). I tried summing them up but I couldn't get to the final value I was expecting

There's surely something I'm fundamentally missing. Can you give me some pointers?

neural-network numpy python
1 Answers

The shape of the input should be (2,1). If it's (1,2) (e.g. can be caused by the following initialization: np.random.randn(2)), the first hidden layer calculation may return an 3x3 matrix because of broadcasting typically implemented in numerical libraries.

r = np.dot(w, a) + b -> (1x3) + (3x1) --> (3x3)

You can't add two vectors of different sizes mathematically, but libraries make it possible to ease calculations. In this case, the result matrix elements would look like following r_{ij}=c_j+d_i where c is the first summand and d is the second summand. This operation is called broadcasting. The smaller dimension is broadcasted across the corresponding one in the other summand, which leaves us with a 3x3 matrix.

It is why the diagonal elements are the ones you're looking for since r_{ii}=c_i+d_i, which are the results you actually propagate to the next layer.

With these dimensions mismatches, it's quite expected that the output layer dimensions are wrong as well.

gunes_28430270
Related