v0.5.0

AdvancedVI v0.5.0

Diff since v0.4.1

v0.5 introduces major breaking changes to the interface of AdvancedVI along tweaks to the default configurations. The changes are summarized below.

Default Configuration Changes

The default parameters for the parameter-free optimizers DoG and DoWG has been changed.
Now, the choice of parameter should be more invariant to dimension such that convergence will become faster than before on high dimensional problems.

The default value of the operator keyword argument of KLMinRepGradDescent has been changed to IdentityOperator from ClipScale. This means that for variational families <:MvLocationScale, optimization may fail since there is nothing enforcing the scale matrix to be positive definite.
Therefore, in case a variational family of <:MvLocationScale is used in combination with IdentityOperator, a warning message instruting to use ClipScale will be displayed.

Interface Changes

An additional layer of indirection, AbstractVariationalAlgorithms has been added.
Previously, all variational inference algorithms were assumed to run SGD in parameter space.
This design however, has proved to be too rigid.
Instead, each algorithm is now assumed to implement three simple interfaces: init, step, and output.
As a result, the signature of optimize had to be revised, but is now simpler, as most configurations will reside as a field to the <:AbstractVariationalAlgorithms object.

A new specialization of estimate_objective have been added that takes the variational algorithm alg as an argument.
Therefore, each algorithm should now implement estimate_objective.
This will automatically choose the right strategy for estimating the associated objective without having to worry about internal implementation details.

Internal Changes

The state of the objectives state may now use a concrete type.
Therefore, to be able to dispatch based on the type of state while avoiding type ambiguities, the state argument in estimate_gradient! has been moved to the front.

Under the new interface AbstractVariationalAlgorithms, the algorithms running SGD in parameter space, currently KLMinRepGradDescent, KLMinRepGradProxDescent, KLMinScoreGradDescent, are treated as distinct algorithms.
However, they all implicitly share the same step function in src/algorithms/common.jl and the same fields for the state object.
This may change in the future.

Merged pull requests:

• Add additional layer of indirection AbstractAlgorithm (#179) (@Red-Portal)
• CompatHelper: bump compat for AdvancedVI to 0.4 for package bench, (keep existing compat) (#181) (@github-actions[bot])
• CompatHelper: bump compat for AdvancedVI to 0.4 for package docs, (keep existing compat) (#182) (@github-actions[bot])
• Make problem as part of the state of ParamSpaceSGD instead of an argument of estimate_gradient! (#185) (@Red-Portal)
• Support Mixing AD Frameworks for LogDensityProblems and the objective (cleaned-up) (#187) (@Red-Portal)
• Split AD tests into workflows (#190) (@Red-Portal)
• Stop testing on DistributionsAD (#191) (@Red-Portal)
• Fix bug in adtype check for Mixed-AD RepGradELBO (#192) (@Red-Portal)
• update the example in the README (#193) (@Red-Portal)
• Create docs/src/tutorials/ directory (#194) (@Red-Portal)
• Add some basic tutorials (#195) (@Red-Portal)
• Use concrete types for objective states (#196) (@Red-Portal)
• Add minibatch subsampling (doubly stochastic) objective (2025 ver.) (#197) (@Red-Portal)
• Fix coverage to aggregate results for all worfklows (#198) (@Red-Portal)
• Update the default parameter value for DoG and DoWG. (#199) (@Red-Portal)
• Move q_init argument of init to the front (#200) (@Red-Portal)
• Make IdentityOperator default for KLMinRepGradDescent (#201) (@Red-Portal)
• Fix typos and improve clarity in README, docs and docstrings (#202) (@Copilot)
• add missing Zygote integration (#203) (@Red-Portal)
• Rename AbstractAlgorithm to AbstractVariationalAlgorithm (#204) (@Red-Portal)
• Remove the type ParamSpaceSGD (#205) (@Red-Portal)
• CompatHelper: bump compat for JSON to 1 for package docs, (keep existing compat) (#206) (@github-actions[bot])
• add specialization of estimate_objective for algorithms (#207) (@Red-Portal)

Closed issues:

• Minibatches (#38)
• Understanding the AdvancedVI interface documented in the README (#173)
• Feature request: (re)start from a given position (#174)
• Rename AbstractAlgorithm to AbstractVI (#183)
• ParamSpaceSGD is not a standard terminology (#184)