Warm start to speed up finite difference at minimum point

[a-jp]

Hi, I'm using the newer interface within CppAD to call IPOPT. I need to obtain finite differences at the minimum point. So after I have called IPOPT via CppAD to obtain the minimum solution, I call it twice more, once at a perturbed +eps and perturbed -eps to compute a numerical central finite difference of the solution at the minimum. Since the perturbations are small, from the first solution, I had expected the second two calls to form the finite difference to be quicker, but on inspection of the iterations from IPOPT I see they're basically the same number of iterations, and I effectively call three full optimisation solutions to obtain my derivatives.

To get round this, it would be ideal to call IPOPT as a warm-start for the last two calls that are at perturbed solutions, based on the data from the first call. What would be the best way to explore this at the moment?

My first thought was that currently we're exposed the IPOPT via a function as per the below, albeit behind the scenes a class is built. Would it be sensible to break this function call into the instantiation of a class that we can call solve on, and then from there, the two further solve calls can be called on the class as warm starts?

All of the above is also called in an outer loop for my problem. At the moment within that loop the interface to IPOPT, the function call:

CppAD::ipopt::solve<DVector, Class>(
                options, x_i, x_l, x_u, g_l, g_u, *this, soln);

Is constructed on every iteration of the loop, and the loop is large. With an eye on the change suggested above to allow warm starts, an added benefit of not accessing IPOPT via the above function and rather an instantiated class, is that if the use case is correct, I would not need to keep creating and destroying the object within the loop and an additional overhead could be saved.

Could you give some comments to the above to understand, primarily, how to get after warm starts, but also more generally how to interface via CppAD to IPOPT but via a class to minimise repeating work if calling IPOPT many times in a loop?

Many thanks

[bradbell]

It sounds like you are trying to compute derivatives of the optimal solution. Since the derivative of the original objective is zero at the solution, perhaps the following method would apply:
https://www.tandfonline.com/doi/full/10.1080/10556788.2017.1406936

[a-jp]

@bradbell thanks! Is that something that's implemented in CppAD that I just call from a user perspective? Do you have any advice when I'm starting from the point of already having working code that makes use of:

CppAD::ipopt::solve<DVector, Class>(
                options, x_i, x_l, x_u, g_l, g_u, *this, soln);

How do I augment this to take advantage of your work on this?

[bradbell]

Well, it is implemented in the following package for a particular problem:
https://cppad-mixed.readthedocs.io/latest/theory.html

[a-jp]

Thanks. Not sure I can meaningfully adopt this or hope to implement your paper.

Can we return to the questions in my original approach. I have an alternate code to compare result against and the finite difference solution I currently use provides identical results to the other code. Can you help me with the questions then, warm start, and breaking the current function call api into a class that can be stored outside a loop when the call to the function-based-api is within a loop? Details in first post. Thanks

[bradbell]

As a first step toward your goal (In the long run, perhaps you can improve ipopt::solve), can you create a modified version of
https://github.com/coin-or/CppAD/blob/master/include/cppad/ipopt/solve.hpp
that returns cppad_nlp and then use it do what you want ?

You will need to extend cppad_nlp (some ways) so that you can get a new startiing point; see get_starting_point in
https://github.com/coin-or/CppAD/blob/master/include/cppad/ipopt/solve_callback.hpp
Once you are able to change the starting point and re-run, perhaps you can extend it to do a warm start ?

By the way, are you differentiating the original objective (that is optimized) or a different objective.

[a-jp]

Ok thanks, I'll take a look at that. Might have some follow ups....

When you say "By the way, are you differentiating the original objective (that is optimized) or a different objective", what do you mean sorry?

[bradbell]

Here is what I did for warm starting cppad_mixed:
https://github.com/bradbell/cppad_mixed/blob/master/src/ipopt_fixed/get_starting_point.cpp

[a-jp]

Ok thanks,

[a-jp]

Bit concerned I'm missing something subtle - what do you mean by: "By the way, are you differentiating the original objective (that is optimized) or a different objective"?

[bradbell]

Sometimes the solution to an optimization problem becomes a parameter in a different optimization problem. Then there are two objectives to talk about the derivative of. See bi-level optimization
https://en.wikipedia.org/wiki/Bilevel_optimization
Note that, for unconstrained problems, the derivative of the inside objective is zero at the solution.

[a-jp]

Sometimes the solution to an optimization problem becomes a parameter in a different optimization problem. Then there are two objectives to talk about the derivative of. See bi-level optimization https://en.wikipedia.org/wiki/Bilevel_optimization Note that, for unconstrained problems, the derivative of the inside objective is zero at the solution.

Yep, ok, so no I'm not doing that. I just want the derivatives at the optimal point.

For the loop I was referring to, I'm simply calling the same object with many different starting conditions and input parameters within the loop (just different doubles passed to the code that control the problem, not 'parameters' in any ipopt sense).

[bradbell]

Is it first or second derivatives that you are interested in ? The derivative at the optimal point is zero (not counting constraints).

[a-jp]

Just the first derivatives at the optimal point.