Weight initialization in neural networks

user_8957025 0
Viewed 406

Hi I am developing a neural network model using keras.

code


def base_model():
    # Initialising the ANN
    regressor = Sequential()   
    # Adding the input layer and the first hidden layer
    regressor.add(Dense(units = 4, kernel_initializer = 'he_normal', activation = 'relu', input_dim = 7))   
    # Adding the second hidden layer
    regressor.add(Dense(units = 2, kernel_initializer = 'he_normal', activation = 'relu'))   
    # Adding the output layer
    regressor.add(Dense(units = 1, kernel_initializer = 'he_normal'))
    # Compiling the ANN
    regressor.compile(optimizer = 'adam', loss = 'mse', metrics = ['mae'])
    return regressor

I have been reading about which kernel_initializer to use and came across the link- https://towardsdatascience.com/hyper-parameters-in-action-part-ii-weight-initializers-35aee1a28404

it talks about glorot and he initializations. I have tried with different intilizations for weights, but all of them give the same results. I want to understand how important is it do a proper initialization?

Thanks

keras machine-learning neural-network initialization
1 Answers

I'll give you an explanation of how much weights initialisation is important.

Let's suppose our NN has an input layer with 1000 neurons, and suppose we start to initialise weights as they are normal distributed with mean 0 and variance 1 ().

At the second layer, we assume that only 500 first layer's neurons are activated, while the other 500 not.

The neuron's input of the second layer z will be the sum of :

so, it will be even normal distributed but with variance .

This means its value will be |z| >> 1 or |z| << 1, so neurons will saturate. The network will learn slowly at all.

A solution is to initialise weights as where is the number of the inputs of the first layer. In this way z will be and so less spreader, consequently neurons are less prone to saturate.

This trick can help as a start but in deep neural networks, due to the presence of hidden multi-layers, the weights initialisation should be done at each layer. A method may be using the batch normalization

Besides this from your code I can see you'v chosen as cost function the MSE, so it is a quadratic cost function. I don't know if your problem is a classification one, but if this is the case I suggest you to use a cross-entropy function as cost function for increasing the learning rate of your network.

user_121129640
Related