I have a terminological question. Is the word "algorithm" synonymous for "model" in the context of statistics? For example, when I fit a generalized linear model with variables and adjust the model parameters for a specific purpose, can I report that I "developed an algorithm" or would this be a false statement? If so, what best describes what I have done? Developed a model/fitted a model/built a model...?
1 Answers
In statistical terminology, algorithms and models are different. Consider the following linear regression model: y = a + b x + epsilon with
- y: dependent variable,
- a: intercept,
- b: slope coefficient,
- x: (univariate) explanatory variable,
- epsilon: the error term, say normally distributed with mean zero and standard deviation sigma.
This is a statistical model because it specifies how we assume the data to have been generated (note: we might be wrong and be using a misspecified model). This model is also a parametric model because apart from three parameters (a, b, and sigma) we know the full distribution of the data.
The algorithmic part starts as soon as we start looking for good parameter values. That is, we optimise some function of the parameters (e.g. maximum likelihood, least squares, etc.). Here we need the algorithm. For ordinary least squares (= maximum likelihood for the linear regression model above), the solution is easy and merely requires some linear algebra. However, more complicated models easily require sophisticated algorithmic implementations. Four examples:
- Nonlinear models estimated by maximum likelihood when the likelihood function needs to be optimised numerically.
- Various gradient descent algorithms to train neural networks.
- The LARS algorithm to calculate the LASSO estimator.
- The coordinate descent algorithm to calculate the LASSO estimator.
The last two examples also show that different algorithms might be used to calculate parameter estimates for the same model (say we used a LASSO estimator for a high-dimensional linear regression).
Finally, there is the fitted model. This is the model specification after having replaced the unknown parameters by the parameter values that have been determined based on the data.
Related