Nonlinear Exponential Regression with Tensorflow.js

Question

I am trying to fit exponential data to an exponential regression with tensorflow.js, such as:

y(x) = c0e^(kx)

I have followed examples where they have fitted a linear regression with just a few epochs, like here.

The problem is that when I change the tensor equation to an exponential function, even if I increase to 500-5000 epochs and providing close initial values, it does not fit properly. With a large learning rate, the variables go to very high values, and low learning rate the variables don't substantially change.

Is there anything I am doing wrong in the code? Is it because optimization is not fit for exponential functions? Is there any other way to do implement this in a browser without using tf.js?

The code I have used is:

const x = tf.tensor1d([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]);
const y = tf.tensor1d([2.5879,3.1153,3.7041,4.6216,5.2307,5.6205,6.9904,7.8416,9.0201,10.5586,12.1638,14.1438,16.5961,19.2497,22.3430]);


const c0 = tf.scalar(2).variable();
const k = tf.scalar(0.10).variable();

// y = c0*e^(k*x)
const fun = (x) => x.mul(k).exp().mul(c0);
const cost = (pred, label) => pred.sub(label).square().mean();

const learning_rate = 0.001;
const optimizer = tf.train.sgd(learning_rate);

// Train the model.
for (let i = 0; i < 20; i++) {
    optimizer.minimize(() => cost(fun(x), y));
}

console.log(`c0: ${c0.dataSync()}, k: ${k.dataSync()}`);

const preds = fun(x).dataSync();
preds.forEach((pred, i) => {
   console.log(`x: ${i}, pred: ${pred}`);
});

user_5069957 · Answer

It has to do with the optimizer used and the set of initial value that might not be converging. It they are not set correctly the model can diverge.

const x = tf.tensor1d([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]);
const y = tf.tensor1d([2.5879,3.1153,3.7041,4.6216,5.2307,5.6205,6.9904,7.8416,9.0201,10.5586,12.1638,14.1438,16.5961,19.2497,22.3430]);


const c0 = tf.scalar(2).variable();
const k = tf.scalar(0.10).variable();

// y = c0*e^(k*x)
const fun = (x) => x.mul(k).exp().mul(c0);
const cost = (pred, label) => pred.sub(label).square().mean();

const learning_rate = 0.1;
const optimizer = tf.train.adagrad(learning_rate);

// Train the model.
for (let i = 0; i < 500; i++) {
    optimizer.minimize(() => cost(fun(x), y));
}

console.log(`c0: ${c0.dataSync()}, k: ${k.dataSync()}`);
fun(x).print()

// [2.4752154, 2.899802, 3.3972201, 3.9799631, 4.6626663, 5.4624777, 6.3994851, 7.4972224, 8.7832594, 10.289897, 12.0549774, 14.1228304, 16.545393, 19.3835087, 22.7084637]

Using tf.train.adagrad, it seems that we have a good convergence. For the initializing value, we can also use Math.Random() just for random value and run the similation many times until we find the set of values that lead to a better prediction. Likewise the learning_rate and the number of epochs used can be fine tuned

Additionnally, it is possibly to use several optimizers and see which one performs best in overall

Nonlinear Exponential Regression with Tensorflow.js

1 Answers