Add dual cone optimizer

[stevengogogo]

The submodule Dual-Con-Gradient-Descent is added in optim_dualcone. The main file we need is dcgd.py that contains DCGD optimizer.

Function API

The Dual Cone method is basically a wrapper of an optimizer. There are three thing need to decide to formulate the Dual-Cone-Optimizer. The only different thing is we need to return PDE loss and boundary loss explicitly in loss function (rather than summing up). Suppose we have multiple boundary conditions, this becomes

loss_fn(data) -> [loss_pde, loss_b1, loss_b2,...]

where loss_i is a scalar. For the above example, we have num_pde=1 because only one term is included.

1. optimizer: This can be Adam or any general torch optimizer
2. num_pde: number of loss terms for first section, the rest of loss terms are considered as boundary conditions.
3. type: This can be center, avg, proj
   

class DCGD(torch.optim.Optimizer):


    def __init__(self, optimizer, num_pde, type='center'):
        '''
        num_pde - the number of the PDEs (boundary conditions excluded)
        type - proj, avg, 
        '''

Idea

• Since DCGD is a torch.optim.Optimizer. It is possible to "cascading" multiple loss terms

Reference:

• https://arxiv.org/pdf/2409.18426

[stevengogogo]

@liruipeng
I suggest we need to merge #11 before we proceed to modify loss function for experiments.

[stevengogogo]

Here is more details about type of dual-cone optimizer: