Neural Net - trying to predict that 5 + 5 = 10

Question

I'm learning about Neural Networks and I recently had this idea: trying to give a NN training data of the function $f(x) = 2x$. The question is, can the NN accurately predict that it has to double the input number to give the correct output?

This is just a "mental exercise", to better my understanding of how NNs work.

My Python code doesn't work, here's what I've tried:

Neural Network class:

import numpy as np

class NeuralNetwork:
    def __init__(self, inputnodes, hiddennodes, outputnodes, learningrate):
        self.inodes = inputnodes
        self.hnodes = hiddennodes
        self.onodes = outputnodes

        self.lr = learningrate

        self.wih = np.random.normal(0.0, pow(self.inodes, -0.5), (self.hnodes, self.inodes))
        self.who = np.random.normal(0.0, pow(self.hnodes, -0.5), (self.onodes, self.hnodes))

    def train(self, inputs_list, targets_list):
        inputs = np.array(inputs_list, ndmin=2).T
        targets = np.array(targets_list, ndmin=2).T

        hidden_outputs = np.dot(self.wih, inputs)
        final_outputs = np.dot(self.who, hidden_outputs)

        output_errors = targets - final_outputs
        hidden_errors = np.dot(self.who.T, output_errors)

        self.who += self.lr * np.dot(
            (output_errors * final_outputs * (1.0 - final_outputs)),
            np.transpose(hidden_outputs)
        )

        self.wih += self.lr * np.dot(
            (hidden_errors * hidden_outputs * (1.0 - hidden_outputs)),
            np.transpose(inputs)
        )

    def query(self, inputs_list):
        inputs = np.array(inputs_list, ndmin=2).T
        hidden_outputs = np.dot(self.wih, inputs)
        final_outputs = np.dot(self.who, hidden_outputs)

        return final_outputs

Training the network and predicting a value:

input_nodes = 1
hidden_nodes = 20
output_nodes = 1

learning_rate = 0.3

nn = NeuralNetwork(input_nodes, hidden_nodes, output_nodes, learning_rate)

for i in range(10):
    i += 1
    inputs = np.log(i)
    targets = np.log(2*i)
    nn.train(inputs, targets)

print(nn.query(np.asfarray([4])))

Here's the output I'm getting trying to run this code:

x.py:26: RuntimeWarning: overflow encountered in multiply
  (output_errors * final_outputs * (1.0 - final_outputs)),  
x.py:31: RuntimeWarning: overflow encountered in multiply
  (hidden_errors * hidden_outputs * (1.0 - hidden_outputs)),  
[[nan]]

I don't really know how to interpret this, and if my design is correct for this application. Any help would be appreciated.

Thanks.

user_2482661 · Answer

Some suggestions:

Since the function of interest (f(x)=2x) is linear and requires only one weight, we can vastly simplify the network by having 1 weight and 0 hidden layers. We're trying to debug a problem, so we should simplify as much as possible to eliminate sources of error. Using a hidden layer with multiple hidden nodes implies that we need to find matrices such that W1.dot(W2)=2 because we seek the function x.dot(W1).dot(W2), which is harder because changing 1 weight changes the entire product; finding the correct answer requires aligning all of those weights.
Because the function of interest is linear, we know that any use of nonlinear functions is a distraction. Also, Saturation of sigmoid and tanh functions, or the dying ReLU phenomenon, could introduce additional problems to the optimization dynamics which could prevent us from making progress. See: https://stats.stackexchange.com/questions/301285/what-is-vanishing-gradient
The learning rate is probably too large. I believe this is the problem because you're having numerical overflow; this can happen when the optimizer consistently overshoots the minimum. See: https://stats.stackexchange.com/questions/364360/how-can-change-in-cost-function-be-positive
Scaling the inputs and the targets of a regression problem can dramatically improve the optimizer dynamics. For an example, see https://stats.stackexchange.com/questions/432707/alternating-negative-and-positive-value-of-slope-and-y-intercept-in-gradient-des/432714#432714
Additional tips for training neural networks are here: https://stats.stackexchange.com/questions/352036/what-should-i-do-when-my-neural-network-doesnt-learn/352037#352037

user_4055723 · Answer

I think you are missing a very important part / building block in artificial neural networks architecture , this block is called the activation function , which tries to normalize output between [0,1] or [-1,1] so i think attaching (which is very important) an activation function after computing every hidden layer outputs may solve this problem , as data propagating network will maintain normalized values for example between [0,1] so overflow may will not happen

notes

sigmoid activation and tanh are most popular and suitable for you problem
your learning rate maybe slightly large , try use 0.01

notes

sigmoid activation and tanh are most popular and suitable for you problem
your learning rate maybe slightly large , try use 0.01

Neural Net - trying to predict that 5 + 5 = 10

2 Answers