Choice of regularization

[david-cortes]

Usually, papers about optimization of GLMs create problem varieties according to regularizations (low-optimal-high).

For example, the papers behind LibLinear first obtain the optimal amount of regularization by cross validation, and focus their benchmarks on timings for those values, then trying out high and low regularizations as varieties (since different solvers have an easier or harder time with more/less regularization).

Example:
https://flore.unifi.it/bitstream/2158/1221395/2/tncg_supplement.pdf

Would be nice to offer a fuller benchmark with more variety of cases beyond just data shapes, with regularization values in particular being one important aspect that affects solver convergence.

[tomMoral]

Hello,

Thanks for the feedback, I definitely agree that this is an important aspect of the benchmark.

The regularization parameter can be changed in the objective function by calling the benchmark with

benchopt run . -o "*[lmbd=...]"

So this can typically be customized in the config for the run.

That being said, if there is some estimate of the best hyper-parameter, we could rescale the benchmark to take regularisation values that are adapted with each dataset.

For instance, adding in set_data(..., best_reg=None), and if it is provided for a given dataset, use lmbd=0.1, 1, 10 for values close to the best one?
Would that make sense or do you see a better way?

[david-cortes]

if there is some estimate of the best hyper-parameter, we could rescale the benchmark to take regularisation values that are adapted with each dataset.

It should be a matter of running a k-fold validation over a reasonable grid and seeing which one works best for each dataset, then hard-coding it in the configs.

Would that make sense or do you see a better way?

I'm not very familiar with how this repository is expected to be used, so cannot comment on the best way to do it, but would be ideal if running the default config would involve different regularizations. Perhaps adding a new parameter with a different name than 'lmbd' and taking string options 'low'/'optimal'/'high' could be an option.

[tomMoral]

Yeah I agree that running a cross-validation is a good way to select interesting lambda parameters.

A cheaper way to scale lmbd is to compute the smoothness constant for the logistic loss and then rescale lmbd with it, as it is done in https://github.com/benchopt/benchmark_logreg_l2/blob/main/objective.py#L32
That way, with lmbd=[0.1, 1, 10] we span problems where the l_2 term is dominated, equal, or dominant.
WDYT about this way of choosing regularisation?

Note that we could have an extra value called "best", which would use the best choice based on CV to complement.

[david-cortes]

Yeah I agree that running a cross-validation is a good way to select interesting lambda parameters.

A cheaper way to scale lmbd is to compute the smoothness constant for the logistic loss and then rescale lmbd with it, as it is done in https://github.com/benchopt/benchmark_logreg_l2/blob/main/objective.py#L32 That way, with lmbd=[0.1, 1, 10] we span problems where the l_2 term is dominated, equal, or dominant. WDYT about this way of choosing regularisation?

Note that we could have an extra value called "best", which would use the best choice based on CV to complement.

Yes, that'd be the ideal thing to do either way.

But it might be the case that all of those arbitrary values are cases where the L2 term is dominated or the other way around (not certain that the number calculated there without looking at 'y' would always be suitable), since the rest of the function is highly dependent on the specific data.

It nevertheless sounds like an improvement over the current situation.