Stochastic parallel gradient descent (SPGD) Algorithm does not always converge when it should

Viewed 38

I'm trying to make an algorithm that minimizes (if possible) the function given in parameter. The method i have to use is the SPGD : Stochastic Parallel Gradient Descent.

I'm using these functions in order to test my algorithm : https://en.wikipedia.org/wiki/Test_functions_for_optimization

The problem i have is that for some of them (The GoldsteinPrice & Beale ones), the algorithm does not converge, when it should.

Here's an exemple : Goldstein function exemple

I should have something that look like this : Three Hump Camel exemple

The code is pretty short :

def func(tab, fonction):
    x = tab[0]
    y = tab[1]
    res = fonction
    return res(x, y)

def generate_variations(m):
    l = len(m)
    mean = 0
    variance = 1
    return np.array([random.gauss(mean, math.sqrt(variance)) for i in range(l)])

def SPGD(start, nb_iter, step, fonction, arr):
    for _ in range(nb_iter):
        all = generate_variations(start)
        pos = func(np.add(start,all), fonction)
        neg = func(np.subtract(start, all), fonction)
        sume = np.subtract(pos, neg)
        sume = np.multiply(all, sume)
        sume = np.multiply(step, sume)
        start = np.subtract(start, sume)
        arr = err(arr, func(start, fonction), _, fonction)
return [start[0], start[1], func(start, fonction)]

Start is numpy array containing the starting coordinates, nb_iter is the number of iterations, step is the algorithm step and fonction is a lambda function that the algorithm will try to minimize. The arr parameter is used for ploting and has no role in the computation.

The algorithm steps are based from this article : https://opg.optica.org/ol/abstract.cfm?uri=ol-34-19-2939

0 Answers
Related