Approximating an exponential fit with a simple neural network

dangom 0
Viewed 37

I've been trying to train a network to solve exponential fits of the form s(t) = s0 * e^(-t/decay_constant). As input, the net takes s and t, and as output it should return s0 and the decay_constant. This seems like a sufficiently simple problem that I would expect a net would be able to satisfactorely approximate.

However, I cannot get it to work. The loss does go down, and the results don't look completely random, but they are definitely worse than what I simple log-linear fit could achieve.

My setup was (takes about 20s on CPU)

  1. A dense net with ReLU activations
  2. Loss based on log-linear LSQ
  3. Train on random exponential decay examples
import torch

net = torch.nn.Sequential(
    torch.nn.Linear(10, 64), torch.nn.ReLU(),
    torch.nn.Linear(64, 64), torch.nn.ReLU(),
    torch.nn.Linear(64, 32), torch.nn.ReLU(),
    torch.nn.Linear(32, 2), torch.nn.ReLU(),  # Want outputs to always be positive.
)

loss = torch.nn.MSELoss()
optimizer = torch.optim.Adam(net.parameters(), lr=0.005)

def signal_model(x, s0, decay):
    return s0 * torch.exp(-x / decay)

# Generate some datapoints
batch_size = 4096

for episode in range(1000):
    # Generate random time, decay_constant s0.
    t = t = (torch.arange(5, 55, 10.) + (torch.rand(batch_size, 5) * 10)).T
    decay_constant = torch.rand(batch_size) * 70 + 10
    s0 = (torch.rand(batch_size) * 2 - 1) * 50 + 100

    # Generate random input data
    y = signal_model(t, s0, decay_constant)
    data = torch.vstack((t, y)).T

    # Make a prediction and calculate loss
    coefficients = net(data)
    l = loss(-torch.log(signal_model(t, *coefficients.T) + 1e-20), -torch.log(y + 1e-20))

    optimizer.zero_grad(); l.backward(); optimizer.step()

To visualize results

import matplotlib.pyplot as plt

t_ = torch.tensor([5, 15, 25, 35, 45], dtype=torch.float32)
plotgrid = torch.arange(0, 50, 0.1)
s = signal_model(t_, 20, 30)

fig, ax = plt.subplots()
ax.plot(plotgrid, signal_model(plotgrid, 20, 30).detach().numpy())
ax.scatter(t_, s.detach().numpy(), label="True")
ax.scatter(
    t_,
    signal_model(t_, *net(torch.hstack((t_, s)).T)).detach().numpy(),
    label="Predicted",
)
ax.legend()

fit

I was wondering whether anyone has insights on why this doesn't work. Is there some fundamental limitation to approximating simple functions? Or is there something that jumps to the eye as inherently wrong? Would love any insights.

What didn't work:

  • tweaking learning rate
  • tweaking number of layers / units per layer
  • tweaking batch_size
  • changing loss to MSE on coefficients or on the signal directly without log
  • regularizing with L1loss
  • using always the same t as input

Somewhat related question: https://stats.stackexchange.com/questions/379884/why-cant-a-single-relu-learn-a-relu

curve-fitting neural-network machine-learning pytorch python
1 Answers

s0 and decay_constant are not fixed parameter values in your generated data. There is not a single true parameter vector for the model to converge to. You might think that giving it a variety of different examples of exponential outputs from different parameters means it is being trained to predict proper exponential fit coefficients from any given dataset, but that's wrong. Rather it is just being trained to reproduce a specific distribution of exponential fit coefficients (what looks like normally distributed data with mean 10 and variance of 490 for decay_constant and mean 100 with variance of 2500 for s0). As a thought experiment, what if we fed in an example data set with parameters way way way far out in the tails of those distributions? The model couldn't be adequate at guessing that relationship given that that part of the (s0, decay_constant) space would be essentially missing entirely from the randomized training set. Think of the space of all possible exponential curves ...

The key thing you are missing is that you can't make a single model that predicts the coefficients for any possible dataset. Rather, you can train the model on any dataset to predict the coefficients for that dataset. If the training data is generated from a sample where those coefficients are fixed, then the model's loss optimization will cause the predicted coefficients to converge to the true coefficients for that dataset. You'd essentially be hijacking a NN optimizer to do what a more common optimizer would do with just the deterministic equation solving you'd normally do.

But if you hand a dataset with no fixed parameters, the outputs can at best represent the distribution of parameters according to however you defined it when making the synthetic training data. Because it's not possible to make a training set that adequately includes every possible exponential curve there can be, the model will perform poorly in general. It may perform OK for curve fitting where the parameters are close to the synthetic training distribution you made, but this will depend a lot of the variance and the quality of convergence.

I made a similar type of model for linear regression coefficient predictions. Note: not fitting a linear model, rather jumping straight to predicting the coefficients of a linear model from a dataset generated by a fixed, true, set of coefficients. It is very similar in spirit to what you're trying to do here, but note how it must be trained on the one single dataset you want the coefficients for. It cannot predict generic coefficients across all possible linear relationships. To get meaningful outputs, you don't just make single predictions - rather you would do something like average the predictions across a test set held out from the same data generated by the fixed, true parameters.

ely0
Related