scipy.optimize.minimize with BFGS: Objective called twice with same parameter vector

user_885287 0
Viewed 498

I'm using scipy.optimize.minimize with method='bfgs' to train a convex objective.

Every time I run a minimization, the first two calls the BFGS optimizer makes to my objective function always have the same parameter vector. This seems unnecessary as it wastes a good few minutes re-calculating the same thing twice.

Minimum Working Example (with a much simpler objective);

from scipy.optimize import minimize

def obj_jac(x):
    """Return objective value and jacobian value wrt. x"""
    print(x)
    return 10*x**2, 20*x

minimize(obj_jac, -100, method='bfgs', jac=True, tol=1e-7)

Output;

[-100.]
[-100.]
[-98.99]
[-94.95]
[-78.79]
[-30.17904355]
[-3.55271368e-15]

Does anyone know if this is expected behaviour for the BFGS implementation in scipy?

Update: I have submitted this as issue #10385 on the Scipy project.

scipy-optimize-minimize numerical-methods scipy optimization python
1 Answers

This is not an expected behaviour or, at least, there is a reporting bug.

By doing the statistics output for the optimization via options parameters:

minimize(obj_jac, -100, method='bfgs', jac=True, tol=1e-7, options={'disp': True})

SciPy outputs the following:

[-100.]
[-100.]
[-98.99]
[-94.95]
[-78.79]
[-30.17904355]
[-3.55271368e-15]
Optimization terminated successfully.
         Current function value: 0.000000
         Iterations: 3
         Function evaluations: 6
         Gradient evaluations: 6

where the reported number of functions and gradient evaluations certainly are off by one. So, there is certainly a stat reporting bug in SciPy for BFGS.

I would also suspect, that there is an inefficiency inside SciPy, which would be along the following lines. Before the iteration loop, evaluate the function and its gradient. Then start the loop and start from evaluating the function and its gradient. That would add an additional function evaluation for the 0th iteration and certainly can be avoided by slight code reorganization (probably, with some trade-off wrt algorithm readability flow).

The following is relevant:

Not being an expert in SciPy, I would say either an old bug popped up out of nowhere (then it should be reported) or it was never fixed in the first place, despite what I comprehended from the GitHub discussions.

user_24528690
Related