Is Stochastic gradient descent a classifier or an optimizer?

user_2934797 0
Viewed 12933

I am new to Machine Learning and I am trying analyze the classification algorithm for a project of mine. I came across SGDClassifier in sklearn library. But a lot of papers have referred to SGD as an optimization technique. Can someone please explain how is SGDClassifier implemented?

gradient-descent scikit-learn classification machine-learning
4 Answers

Taken from SGD sikit-learn documentation

loss="hinge": (soft-margin) linear Support Vector Machine, loss="modified_huber": smoothed hinge loss, loss="log": logistic regression

user_34514410

SGDClassifier is a linear classifier which implements regularized linear models with stochastic gradient descent (SGD) learning

Other classifiers:

classifiers = [
("ASGD", SGDClassifier(average=True, max_iter=100)),
("Perceptron", Perceptron(tol=1e-3)),
("Passive-Aggressive I", PassiveAggressiveClassifier(loss='hinge',
                                                     C=1.0, tol=1e-4)),
("Passive-Aggressive II", PassiveAggressiveClassifier(loss='squared_hinge',
                                                      C=1.0, tol=1e-4)),
("SAG", LogisticRegression(solver='sag', tol=1e-1, C=1.e4 / X.shape[0]))

]

Stochastic Gradient Descent (sgd) is a solver. It is a simple and efficient approach for discriminative learning of linear classifiers under convex loss functions such as (linear) Support Vector Machines and Logistic Regression.

Other alternative solvers for sgd in neural_network.MLPClassifier are lbfgs and adam

solver : {‘lbfgs’, ‘sgd’, ‘adam’}, default ‘adam’

The solver for weight optimization.

‘lbfgs’ is an optimizer in the family of quasi-Newton methods

‘sgd’ refers to stochastic gradient descent.

‘adam’ refers to a stochastic gradient-based optimizer proposed by Kingma, Diederik, and Jimmy Ba

Details about implementation of SGDClassifier can be read @ SGDClassifier documentation page.

in brief:

This estimator implements regularized linear models with stochastic gradient descent (SGD) learning: the gradient of the loss is estimated each sample at a time and the model is updated along the way with a decreasing strength schedule (aka learning rate). SGD allows minibatch (online/out-of-core) learning

user_49993940

Lets seggregate each word in simple english meaning

Stochastic - Random, Gradient - slope, Descent - downwards

Basically, this technique is used as an "optimizing algorithm" for finding the parameters with minimal convex loss/cost function.

By which we can find out the slope of the line which has minimal loss for linear classifiers i.e. (SVM & Logistic Regression)

Some of the other ways in which it is performed:

  1. Batch Gradient Descent
  2. Stochastic Gradient Descent
  3. Mini-batch

For more details, on the above please go through the referred link: https://www.geeksforgeeks.org/ml-stochastic-gradient-descent-sgd/

user_73914040
Related