Is there a way for a Keras model to predict with precision around 14-17 of significant digits?

Majk 0
Viewed 40

I've created simple Keras model (prediction, not classification or regression)

inpLayer = tf.keras.layers.Input((1,), dtype=tf.float64)
hiddenLayer = tf.keras.layers.Dense(2, activation=tf.atan, dtype=tf.float64)(inpLayer)
outputLayer = tf.keras.layers.Dense(1, activation='linear', dtype=tf.float64)(hiddenLayer)
model = tf.keras.models.Model(inpLayer, outputLayer)

with Adam optimizer and I used Mean squared error as a loss function and using batch (offline) gradient descent. I have a small dataset (2 "rows").

x = [35., 49.]

y = [0., 1.]

Am I able to train this simple Neural Network on those features and labels? By "train" I mean: When the model predicts, it returns output close to the labels using whole float64. So the output should have 15 to 17 significant decimal digits precision.

Acceptable: Output for 0 would be in range 9.9e-14 to 1.0e-17. Output for 1 would be in range 1.0e+14 to 9.9e+17.

Is it even possible to train the Neural network like that or it is not? If so, what am I doing wrong there?

neural-network machine-learning keras tensorflow
1 Answers

You can get as close as you want. Just increase the steps_per_epoch parameter. Additional you can decrease the learning rate during training for better results. For example:

import tensorflow as tf

x = 5000*[[35.], [49.]]
y = 5000*[[0.], [1.]]

inpLayer = tf.keras.layers.Input((1,), dtype=tf.float64)
hiddenLayer = tf.keras.layers.Dense(2, activation=tf.atan, dtype=tf.float64)(inpLayer)
outputLayer = tf.keras.layers.Dense(1, activation='linear', dtype=tf.float64)(hiddenLayer)
model = tf.keras.models.Model(inpLayer, outputLayer)
model.compile(loss='mse', optimizer='adam')
callbacks = [tf.keras.callbacks.LearningRateScheduler(lambda epoch, learning_rate: 0.8 * learning_rate)]
model.fit(x, y, epochs=5, batch_size=2, steps_per_epoch=3000, verbose=1, callbacks=callbacks)

model.evaluate([[35.], [49.]], [[0.], [1.]])
a,b = model.predict([[35.], [49.]])
print(a.astype(str), b.astype(str))


Epoch 1/5
3000/3000 [==============================] - 7s 2ms/step - loss: 0.3457 - lr: 8.0000e-04
Epoch 2/5
3000/3000 [==============================] - 6s 2ms/step - loss: 0.0537 - lr: 6.4000e-04
Epoch 3/5
3000/3000 [==============================] - 6s 2ms/step - loss: 3.7255e-04 - lr: 5.1200e-04
Epoch 4/5
3000/3000 [==============================] - 6s 2ms/step - loss: 1.7512e-09 - lr: 4.0960e-04
Epoch 5/5
3000/3000 [==============================] - 6s 2ms/step - loss: 4.0492e-29 - lr: 3.2768e-04
1/1 [==============================] - 0s 107ms/step - loss: 2.6006e-32
['5.204170427930421e-17'] ['1.0000000000000002']

You can use a generator for your data:

def gen_data():
    x = [[35.], [49.]]
    y = [[0.], [1.]]
    data = x, y
    while True:
        yield data    

dataset = tf.data.Dataset.from_generator(gen_data, output_types=(tf.float32, tf.float64))

Then run the fit function like this:

model.fit(dataset, epochs=5, batch_size=2, steps_per_epoch=3000, verbose=1, callbacks=callbacks)
AndrzejO0
Related