python neural nets explanation

model = Sequential()

We are using the Keras Sequential API to build models. There is the alternative functional API. Sequential API allows to create models layer-by-layer for most problems. It does not allow you to create models that share layers or have multiple inputs or outputs.

nFeatures = X.shape[1]

We get the number of features. X.shape = (No_of_inputs, no_of_features)

Number of inputs will be equal to the number of units/neurons in the input layer.

model.add(Dense(20,
        input_dim=nFeatures,
        activation="relu",
        kernel_initializer="random_normal",
        bias_initializer="zeros"))
    
    

We define the first input layer with nFeatures units along with another dense layer with 20 units. Dense layers have a linear part and a non-linear part. The linear part is the eqn of a line Wx+b where W is the weights matrix and b is the bias vector. The non-linear part would be an activation function which is just a non-linear function which adds that randomness to the weights and these also affect the model convergence speed or the training time to convergence, the loss etc. Using ReLU Rectified Liner Unit function here. Bias & Kernal initializers defines how the bias and kernal/weight matrix should be initialized. Certain heuristic and research evident methods such as Xavier initializer, Glorot normal initializer, random_normal are used to initialize the weight matrix as these help in preventing exploding and vanishing gradients. Bias vector just has to be zeros.

nOutput = y.shape[1]
model.add(Dense(nOutput,
        activation="softmax",
        kernel_initializer="random_normal",
        bias_initializer="zeros"))
    

Define the first input layer with nOutput units. Number of units/neurons in the output layer will depend on the number of classes. Ex: Binary classifier 1 output. Classes 0 or 1. Here this is a multi-class classifier. So number of units/neurons in the output layer = number of classes.

model.compile(optimizer="adam",
        loss="categorical_crossentropy",
        metrics=["categorical_accuracy"])
    
    

We define the optimizer which is the optimization algorithm used to minimize the loss function to converge the network during training. Adam stands for Adaptive Momentum. Gradient descent, Nadam, SGD etc can be used here. This is the algorithm which defines the way in which the gradients are computed to minimize the loss function and these gradients are computed in a chain fashion starting from the output unit back to the layer 1 and the weights and bias matrices are adjusted accordingly. Finally, Loss function is the metric used to evaluate the network at each iteration and to determine how the weights should be adjusted in the iterative training process. The 'metric' argument is usually used to monitor other loss functions/metric. For a classifier here, it can be precision, recall, accuracy etc. We already see the categorical_crossentropy values during training as 'loss' so it won't be passed as a metric.

Now the model.fit function can be called passing the train data (or both train and validation data), epoch count, batch_size etc to start the training.

Hope that helps! Thanks!

At first, in this line:

model = Sequential()

You are using a wrapper class to build a sequential model, a model where every layer output is connected to the input of the next layer. Your model is composed of two layers, the first layer:

• Takes an input of dimension nFeatures.
• Is using a relu function as activation. The REctifier Linear Unit is a function which looks something like this:
  Relu graph
  As for every activation function it is applied to the inner matricial product of the layer: layers_weights @ input + bias
• You can specify how to initialize the weights of the Dense layer, this is done with the kernel_initializer argument, here the random_normal means that every weights will have a normal distribution.
• the bias_initializer is the same idea for the bias term (the term added to the iner matrix multiplication)

Your second layer uses another activation function, e.g. softmax which is useful when you want to predict probability distributions, e.g. for classification tasks.\

The third chunk of code is where you compile your model, to compile it you must specify two arguments loss and optimizer:

• The loss is the function which takes the predictions of your model and the ground truth values, and evaluate the goodness of the predictions. There is a number of them and every loss function has its specificity. You could even define one by yourself. Every loss have one common characteristic, less is better. Here you use the categorical_crossentropy function which is a good loss function to do multi-class classification (I'm assuming that's what you are up to).
• The optimizer argument takes the loss function output. And computes the gradient for every weight/parameter of your model. There is a number of different optimizer, the adam optimizer is a combination of AdaGrad and rmsprop optimizers. Don't worry if those don't mean anything to you. You will find a gentle introduction to the adam optimizer here. Gentle Introduction to the Adam Optimization Algorithm for Deep Learning