How tensorflow initialize weight values in my neural network

Admin 1
Viewed 40

i found that by default keras use Glorot/Xavier to initialize weight, this means that the values will be between +- (sqrt(6 / float(F_in + F_out))

But in my case i use the architecture below, with ishape = (None, 4): i don't use fixed input size. ( My input data is a DNA sequence in one hot encoding)

model = keras.Sequential()
model.add(Conv1D(filternumber, b, activation='relu', input_shape=ishape))
model.add(MaxPooling1D(pool_size=a))
model.add(GlobalAvgPool1D(data_format="channels_last"))
model.add(Dense(32, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))

How keras initialize the weight, considering that it doesn't know the input size? What is the best way to initialize the weight in my case?

Thanks

keras tensorflow python
1 Answers

Your model "knows" your input size in this instance, it is 4. The None in (None, 4) refers to a variable number of vectors which is the batch size, but the input vector size and the layers are what drives the Sequential model's shape, from input to output.

You can read in this Sequential model's developer guide that even if the shape is not specified at all, the model won't initialize any weights until its first inference where the input data will fix the input shape. The model will build its weights from the now known shape.

Any weight initializer which needs knowledge on input and output features size would work in both cases.

I have not much advise on the weight initialization technique to choose, I would consider the weight initialization as an hyperparameter to tune.

user_104090930
Related