I have experimented with neural networks in the past, and I'm now working on a animal picture recognition program.
For this neural network, each node of the input layer (representing a pixel of the sample image) consists of an array with three float values (R, G and B values the pixel), and these values stay separate in the first and second hidden layers, and then get summed up to get the ouput layer.
On the first iteration, the weights are randomized floats between -1 and 1 and the biases are all 0 for now.
I've made a backpropagation algorithm (which, for now, only affects weights and leaves biases at 0) that, at first, seems to work just fine, but after some iterations (about 5) the weights just seem to skyrocket and approximate to infinity (with numbers like ...E+25), before eventually turning to NaN. Other times, pretty much none of the weights in any of the arrays changes by even the slightest bit for dozens of iterazions, and then they quickly skyrocket to eventually reach NaN.
I have also noticed (with the Visual Studio debugger) that when there is a new iteration, only some weights change value (usually by a lot) and everything else remains perfectly still.
The input layer is currently made of 400 nodes (I'm testing this with 20x20 images), but will eventually be made up of 10000 nodes or more, and the hidden layers have 16 nodes each (at first, they had over 200 nodes, but I thought that might have been the cause of the issue so I lowered that to 16, with no change in the result). The output layer is currently made of 2 nodes, but will eventually grow to 10+ nodes. The learning rate is set to 0.1.
The cose I have so far is this:
#region Neural Network
int pixels = image.Width * image.Height;
#region Forward Propagation
int index = 0;
for (int x = 0; x < image.Width; x++)
{
for (int y = 0; y < image.Height; y++)
{
inputLayer[index, 0] = image.GetPixel(x, y).R / 255f;
inputLayer[index, 1] = image.GetPixel(x, y).G / 255f;
inputLayer[index, 2] = image.GetPixel(x, y).B / 255f;
index++;
}
}
for (int i = 0; i < firstHiddenLayer.GetLength(0); i++)
{
for (int color = 0; color < 3; color++)
{
float sum = 0;
for (int input = 0; input < inputLayer.GetLength(0); input++)
{
sum += inputLayer[input, color] * inputLayerWeights[input, i, color];
}
firstHiddenLayer[i, color] = ActivationFunction(sum + firstHiddenLayerBiases[i, color], "ReLU");
}
}
for (int i = 0; i < secondHiddenLayer.GetLength(0); i++)
{
for (int color = 0; color < 3; color++)
{
float sum = 0;
for (int input = 0; input < firstHiddenLayer.GetLength(0); input++)
{
sum += firstHiddenLayer[input, color] * firstHiddenLayerWeights[input, i, color];
}
secondHiddenLayer[i, color] = ActivationFunction(sum + secondHiddenLayerBiases[i, color], "ReLU");
}
}
for (int i = 0; i < outputLayer.Length; i++)
{
float sum = 0;
for (int color = 0; color < 3; color++)
{
for (int input = 0; input < secondHiddenLayer.GetLength(0); input++)
{
sum += secondHiddenLayer[input, color] * secondHiddenLayerWeights[input, i, color];
}
}
outputLayer[i] = ActivationFunction(sum + outputLayerBiases[i], "Sigmoid");
Console.WriteLine(sum);
}
#endregion
#region Backpropagation
float[] errors = new float[outputLayer.Length];
for (int i = 0; i < outputLayer.Length; i++)
{
errors[i] = outputLayer[i] - imageClass == i ? 1 : 0;
}
float[] gammas = new float[outputLayer.Length];
for (int i = 0; i < outputLayer.Length; i++)
{
gammas[i] = errors[i] * (secondHiddenLayer[i, color] > 0 ? 1 : 0);
}
for (int i = 0; i < secondHiddenLayer.GetLength(0); i++)
{
for (int j = 0; j < outputLayer.Length; j++)
{
secondHiddenLayerWeights[i, j, 0] -= learningRate * gammas[j] * secondHiddenLayer[i, 0];
secondHiddenLayerWeights[i, j, 1] -= learningRate * gammas[j] * secondHiddenLayer[i, 1];
secondHiddenLayerWeights[i, j, 2] -= learningRate * gammas[j] * secondHiddenLayer[i, 2];
}
}
float[,] secondHiddenLayerErrors = new float[secondHiddenLayer.GetLength(0), 3];
for (int i = 0; i < secondHiddenLayerErrors.GetLength(0); i++)
{
for (int color = 0; color < 3; color++)
{
float error = 0;
for (int j = 0; j < outputLayer.Length; j++)
{
error += errors[j] * secondHiddenLayerWeights[i, j, color];
}
secondHiddenLayerErrors[i, color] = error;
}
}
float[,] secondHiddenLayerGammas = new float[secondHiddenLayer.GetLength(0), 3];
for (int i = 0; i < secondHiddenLayerErrors.GetLength(0); i++)
{
for (int color = 0; color < 3; color++)
{
secondHiddenLayerGammas[i, color] = secondHiddenLayerErrors[i, color] * (secondHiddenLayer[i, color] > 0 ? 1 : 0);
}
}
for (int i = 0; i < firstHiddenLayer.GetLength(0); i++)
{
for (int j = 0; j < secondHiddenLayer.GetLength(0); j++)
{
firstHiddenLayerWeights[i, j, 0] -= learningRate * secondHiddenLayerGammas[j, 0] * firstHiddenLayer[i, 0];
firstHiddenLayerWeights[i, j, 1] -= learningRate * secondHiddenLayerGammas[j, 1] * firstHiddenLayer[i, 1];
firstHiddenLayerWeights[i, j, 2] -= learningRate * secondHiddenLayerGammas[j, 2] * firstHiddenLayer[i, 2];
}
}
float[,] firstHiddenLayerErrors = new float[firstHiddenLayer.GetLength(0), 3];
for (int i = 0; i < firstHiddenLayerErrors.GetLength(0); i++)
{
for (int color = 0; color < 3; color++)
{
float error = 0;
for (int j = 0; j < secondHiddenLayer.GetLength(0); j++)
{
error += secondHiddenLayerErrors[j, color] * firstHiddenLayerWeights[i, j, color];
}
firstHiddenLayerErrors[i, color] = error;
}
}
float[,] firstHiddenLayerGammas = new float[firstHiddenLayer.GetLength(0), 3];
for (int i = 0; i < firstHiddenLayerErrors.GetLength(0); i++)
{
for (int color = 0; color < 3; color++)
{
firstHiddenLayerGammas[i, color] = firstHiddenLayerErrors[i, color] * (ActivationFunction(firstHiddenLayer[i, color], "Sigmoid") * (1 - ActivationFunction(firstHiddenLayer[i, color], "Sigmoid")));
}
}
for (int i = 0; i < inputLayer.GetLength(0); i++)
{
for (int j = 0; j < firstHiddenLayer.GetLength(0); j++)
{
inputLayerWeights[i, j, 0] -= learningRate * firstHiddenLayerGammas[j, 0] * inputLayer[i, 0];
inputLayerWeights[i, j, 1] -= learningRate * firstHiddenLayerGammas[j, 1] * inputLayer[i, 1];
inputLayerWeights[i, j, 2] -= learningRate * firstHiddenLayerGammas[j, 2] * inputLayer[i, 2];
}
}
(Note that in the code the first "errors" array, as well as the first "gammas" array, are referred to the output layer)
I'm pretty sure the issue is being caused by the error or gamma formulas being wrong, but I'm not sure, so if any of you has any tips to share I would highly aprreciate them.