Re-Factor Interface(s)

[manuelbb-upb]

Many of the current design decisions are inspired by MathOptInterface and -- to a certain degree -- ModelingToolkit/Optimization.jl.
However, I never really carried through with it, which why some things feel awkward, and future extensions won't fit easily.

Multiple Algorithms and Flavors of Algorithms

At the moment, there is only the trust region descent algorithm.
In the near future, I want to include nonlinear CG variants, and we could also think about 2nd degree directions.
The algorithm basically stays the same, so we have different flavors of the same algorithmic framework.
At the time of writing, we would configure the flavors using AlgorithmOptions.

Moreover, the exploration of the Pareto Front constitutes an entirely new algorithm.
This algorithm will have different specific properties (relating to stopping, meta-parameters etc.).
A new configuration object is needed.

This is the main reason for why I think it beneficial to make AlgorithmOptions an optimizer struct for the trust region descent framework and rename it accordingly, e.g., TrustRegionDescentOptimizer.
The exploration algorithm would be configured by something like ExplorationOptimizer.

Stopping Criteria

I am not quite satisfied with how stopping criteria are implemented right now.
In future versions, every (algorithm-specific) stopping criterion will be part of the optimizer config struct.
The AbstractStoppingCriterion interface will get additional enable! and is_enabled functions.
Formalizing the state of positional arguments for evaluate_stopping_criterion at various points in the algorithm is also a ToDo.

Initial Point and Precision

For historic reasons, AlgorithmOptions has many Float64 default values. The constant DEFAULT_PRECISION is Float32 however.
Both float types do not really matter right now, precision is inferred from x0, which is given as a positional argument to optimize.
IMHO, the initial primal values should rather be an optimizer property (thus set in TrustRegionDescentOptimizer).
The current AlgorithmOptions constructors (by means of Paramaters), are not really suited for default type parameters (the float type), so we will have to come up with something else.

Re-Thinking Modelling and Surrogatization

Right now, an AbstractMOPSurrogate is initialized with init_models(mop::AbstractMOP, n_vars, scaler).
Hence, AbstractMOP defines, how surrogate models are constructed.
But surrogatization is not really a property of the problem model mop, but rather of the algorithm, and possibly some additional configuration or caching object (and the scaler perhaps).

With the perspective goal of surrogatizing sub-components of problem graphs, it is maybe best to expose all sub-expressions/sub-functions of an AbstractMOP.
Evaluation and differentiation would then be suspiciously similar to the Nonlinear submodule of MathOptInterface or what is stored away in "src/ToDoSubmodelling".

The custom submodelling module was meant to mimic the nonlinear modelling and evaluation of MathOptInterface.Nonlinear at the time (something about a year ago?).
For user side modelling there are various AbstractNodes to build the computation graph.
These are then mapped to MetaNodes in a DAG.
MetaNodes are similar to Nodes in MOI, with various meta-data and indexing into value or operator arrays.
The DAG is a skeleton for something between a Model and an Evaluator in MOI.

Back then, I chose to make my own DAG, because
a) I am interested in functions rather than expressions,
b) I wished for Multi-In-Multi-Out operators.
c) The MOI nonlinear interface and the vector interface are experimental, and did not support a "vector of functions" previously, something I wanted.
d) I was (and still am) unsure about where to fit the surrogate models.

a) is not really an issue as functions can be registered with MOI.
b) is prohibited as per this line.
However, the graph strucuter might change soon: jump-dev/MathOptInterface.jl#2402

I guess it is easiest to revive the archived DAG stuff and see what happens over at MOI, to not reinvent the wheel.
Besides surrogatization, parallelization is something else to keep in mind, which I can enable more easily within our own structure (at the expense of an additional dimension in value arrays).