Keras: loss keeps increasing

user_6288172 0
Viewed 7050

Code:

import keras
import numpy as np

x = []
y = []

for i in range(1000):
    x.append((i/10.0))
    y.append(2.71828 ** (i/10.0))

x = np.asarray(x)
y = np.asarray(y)
x = x.T
y = y.T

model = keras.models.Sequential()
model.add(keras.layers.Dense(1, input_dim=1, activation='relu'))
model.add(keras.layers.Dense(100, activation='relu'))
model.add(keras.layers.Dense(1))

model.compile(loss='mean_squared_error', optimizer=keras.optimizers.SGD(lr=0.001))
model.fit(x, y, batch_size=1, shuffle=False)

tx = [0.0, 1.0, 10.0]
tx = np.asarray(tx)
tx = tx.T

print(model.predict(tx))

This is a very simple neural net which intends to map e^x. Its my first time using keras and when I run it, the loss keeps increasing to infinity. Instead it should decrease.

keras tensorflow deep-learning neural-network python
1 Answers

If a loss increases to infinity it is a sign that the learning rate is too high. The problem with fitting e^x, where x = 100 as in your case is that the difference in values will be very large. Therefore the gradient is going to be very large and the updates are going to be just as large, depending on your learning rate. However if your learning rate is very small, the network will be dominated by those large updates. That is one of the reasons why values are normalized in deep learning.

One quick solution would be to just use the first 100 values or decrease your step size.

user_39948240
Related