FEAT - Double quadratic datafit (quad-quad)

[tomaszkacprzak]

Description of the feature

In many applications there is a need to have an asymmetric loss that weights negative and positive residuals differently. A good function for this is double quadratic loss (quad-quad):

$L (\epsilon, \alpha) = 2 · ( \alpha + (1 − 2\alpha) · 1[\epsilon > 0] ) · || \epsilon ||_2^2$

where $\epsilon = Xw - y$ is the residual, $\alpha$ controls the degree of asymmetry and $1[·]$ is the indicator function. For $\alpha=0.5$ this function is equivalent to Quadratic datafit.

Would it be possible to add DoubleQuadratic datafit and have it compatible with the Lasso and other penalties?

Considered alternatives

Another possibility is to use Pinball loss.

[mathurinm]

Welcome back @tomaszkacprzak !
$\epsilon$ is a vector, so I'm not sure how $1[\epsilon > 0]$ is to be interpreted, when multiplying $\Vert \epsilon \Vert^2$?

Is it not something like:

$L (\epsilon, \alpha) = 2 · \sum_i ( \alpha + (1 − 2\alpha) · 1[\epsilon_i > 0] ) \epsilon_i^2$

Can you give the implementation a try through a PR, like you did for positive group lasso ?
Thanks a lot

[Badr-MOUFAD]

Also I believe it should be $1 / 2$ instead of $2$ to get ride of the coefficient that pops up after differentiation

$L (\epsilon, \alpha) = \frac{1}{2} \sum_i ( \alpha + (1 − 2\alpha) · 1[\epsilon_i > 0] ) \epsilon_i^2$

[tomaszkacprzak]

Hi @mathurinm thank you for a fast reply. Indeed there was a mistake in my equation, the one you gave should be right. I can try a PR. I could start with a Quadratic and modify all the functions. Do you expect any difficulties for optimisation?

[mathurinm]

Yes that sounds like a plan, and no I don't foresee any difficulty!

[mathurinm]

@tomaszkacprzak could you also please let us know in which context you use skglm, and what it brings you compared to alternatives?

[mathurinm]

Hi @tomaszkacprzak, any news on this?

[Aishwarya0811]

Hi @mathurinm i recently gone through your discussion approach is understandable. I can try a PR

[Aishwarya0811]

Hi @mathurinm @tomaszkacprzak!

I've successfully implemented the DoubleQuadratic datafit as discussed.

Implementation highlights:

• Asymmetric quadratic loss: L(ε,α) = (1/2n) * Σᵢ (2α + 2(1-2α) * 1[εᵢ>0]) * εᵢ²
• When α=0.5, gives identical results to standard Quadratic (verified )
• Support for both dense and sparse matrices
• Full compatibility with existing penalties (L1, MCP, etc.)
• Comprehensive test coverage

Verification:

• α=0.5 matches Quadratic exactly (loss and gradient)
• Integration with AndersonCD solver works
• Asymmetric behavior confirmed for α≠0.5
• All tests pass

I'll submit the PR shortly. Thanks for the clear mathematical specification!