deep neural network model stops learning after one epoch

user_14222379 0
Viewed 1611

I am training a unsupervised NN model and for some reason, after exactly one epoch (80 steps), model stops learning. enter image description here] Do you have any idea why it might happen and what should I do to prevent it?

This is more info about my NN: I have a deep NN that tries to solve an optimization problem. My loss function is customized and it is my objective function in the optimization problem. So if my optimization problems is min f(x) ==> loss, now in my DNN loss = f(x). I have 64 input, 64 output, 3 layers in between :

self.l1 = nn.Linear(input_size, hidden_size)
self.relu1 = nn.LeakyReLU()
self.BN1 = nn.BatchNorm1d(hidden_size)

and last layer is:

self.l5 = nn.Linear(hidden_size, output_size)
self.tan5 = nn.Tanh()
self.BN5 = nn.BatchNorm1d(output_size)

to scale my network. with more layers and nodes(doubles: 8 layers each 200 nodes), I can get a little more progress toward lower error, but again after 100 steps training error becomes flat!

enter image description here

unsupervised-learning pytorch deep-learning machine-learning neural-network
1 Answers

The symptom is that the training loss stops being improved relatively early. Suppose that your problem is learnable at all, there are many reasons for the for this behavior. Following are most relavant:

  • Improper preprocessing of input: Neural network prefers input with zero mean. E.g., if the input is all positive, it will restrict the weights to be updated in the same direction, which may not be desirable (https://youtu.be/gYpoJMlgyXA).

Therefore, you may want to subtract the mean from all the images (e.g., subtract 127.5 from each of the 3 channels). Scaling to make unit standard deviation in each channel may also be helpful.

  • Generalization ability of the network: The network is not complicated or deep enough for the task.

    This is very easy to check. You can train the network on just a few images (says from 3 to 10). The network should be able to overfit the data and drives the loss to almost 0. If it is not the case, you may have to add more layers such as using more than 1 Dense layer.

Another good idea is to used pre-trained weights (in applications of Keras documentation). You may adjust the Dense layers at the top to fit with your problem.

  • Improper weight initialization. Improper weight initialization can prevent the network from converging (https://youtu.be/gYpoJMlgyXA, the same video as before).

    For the ReLU activation, you may want to use He initialization instead of the default Glorot initialiation. I find that this may be necessary sometimes but not always.

Lastly, you can use debugging tools for Keras such as keras-vis, keplr-io, deep-viz-keras. They are very useful to open the blackbox of convolutional networks.

I faced the same problem then I followed the following: After going through a blog post, I managed to determine that my problem resulted from the encoding of my labels. Originally I had them as one-hot encodings which looked like [[0, 1], [1, 0], [1, 0]] and in the blog post they were in the format [0 1 0 0 1]. Changing my labels to this and using binary crossentropy has gotten my model to work properly. Thanks to Ngoc Anh Huynh and rafaelvalle!

user_99206170
Related