Skip to content

Experiment on Dual cone optimizer #24

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 3 commits into
base: stc/add-dual-cone-optimizer
Choose a base branch
from
Draft

Experiment on Dual cone optimizer #24

wants to merge 3 commits into from

Conversation

stevengogogo
Copy link
Collaborator

No description provided.

@stevengogogo stevengogogo marked this pull request as draft June 17, 2025 00:02
@stevengogogo
Copy link
Collaborator Author

stevengogogo commented Jun 17, 2025 •
edited
Loading

Exp:

Code: 5a7fb90
Test condition: #11 (comment)

PINN+hard_bc

 python pinn_1d.py   --levels 1 --epochs 20000 --lr 1e-4 --activation tanh --sweeps 1 --plot --hidden_dims 128 128 128 128 --high_freq 16 --bc_weight 500 --nx 2000  --gamma 1 --loss_type 0 --enforce_bc
Options used:
   --nx: 2000
   --nx_eval: 256
   --num_checks: 20
   --num_plots: 10
   --epochs: 20000
   --adam_epochs: 20000
   --sweeps: 1
   --hidden_dims: [128, 128, 128, 128]
   --ax: 0.0
   --bx: 1.0
   --high_freq: 16
   --gamma: 1.0
   --mu: 70
   --lr: 0.0001
   --levels: 1
   --loss_type: 0
   --activation: tanh
   --plot: True
   --clear: True
   --problem_id: 1
   --enforce_bc: True
   --bc_weight: 500.0
Using loss: PINN Loss
MultiLevelNN(
  (models): ModuleList(
    (0): Level(
      (linear): ModuleList(
        (0): Linear(in_features=1, out_features=128, bias=True)
        (1-3): 3 x Linear(in_features=128, out_features=128, bias=True)
        (4): Linear(in_features=128, out_features=1, bias=True)
      )
      (act): Tanh()
    )
  )
)

Training Sweep 0

Training Level 0
Level 0: TRAIN
Iteration    999/ 20000, PINN Loss: 9.5884e+06, Err 2-norm:  8.6897e+01, inf-norm: 1.1050e+01
Iteration   1999/ 20000, PINN Loss: 8.6899e+06, Err 2-norm:  1.1593e+02, inf-norm: 1.0727e+01
Iteration   2999/ 20000, PINN Loss: 8.0062e+06, Err 2-norm:  1.4202e+02, inf-norm: 1.2087e+01
Iteration   3999/ 20000, PINN Loss: 7.4509e+06, Err 2-norm:  1.6569e+02, inf-norm: 1.4167e+01
Iteration   4999/ 20000, PINN Loss: 6.9887e+06, Err 2-norm:  1.8726e+02, inf-norm: 1.6040e+01
Iteration   5999/ 20000, PINN Loss: 6.5988e+06, Err 2-norm:  2.0692e+02, inf-norm: 1.7736e+01
Iteration   6999/ 20000, PINN Loss: 6.2673e+06, Err 2-norm:  2.2480e+02, inf-norm: 1.9273e+01
Iteration   7999/ 20000, PINN Loss: 5.9839e+06, Err 2-norm:  2.4106e+02, inf-norm: 2.0665e+01
Iteration   8999/ 20000, PINN Loss: 5.7404e+06, Err 2-norm:  2.5580e+02, inf-norm: 2.1925e+01
Iteration   9999/ 20000, PINN Loss: 5.5305e+06, Err 2-norm:  2.6917e+02, inf-norm: 2.3066e+01
Iteration  10999/ 20000, PINN Loss: 5.3490e+06, Err 2-norm:  2.8128e+02, inf-norm: 2.4098e+01
Iteration  11999/ 20000, PINN Loss: 5.1914e+06, Err 2-norm:  2.9223e+02, inf-norm: 2.5041e+01
Iteration  12999/ 20000, PINN Loss: 5.0543e+06, Err 2-norm:  3.0214e+02, inf-norm: 2.5896e+01
Iteration  13999/ 20000, PINN Loss: 4.9345e+06, Err 2-norm:  3.1109e+02, inf-norm: 2.6668e+01
Iteration  14999/ 20000, PINN Loss: 4.8297e+06, Err 2-norm:  3.1917e+02, inf-norm: 2.7365e+01
Iteration  15999/ 20000, PINN Loss: 4.7378e+06, Err 2-norm:  3.2648e+02, inf-norm: 2.7994e+01
Iteration  16999/ 20000, PINN Loss: 4.6570e+06, Err 2-norm:  3.3307e+02, inf-norm: 2.8562e+01
Iteration  17999/ 20000, PINN Loss: 4.5857e+06, Err 2-norm:  3.3903e+02, inf-norm: 2.9075e+01
Iteration  18999/ 20000, PINN Loss: 4.5228e+06, Err 2-norm:  3.4440e+02, inf-norm: 2.9538e+01
Iteration  19999/ 20000, PINN Loss: 4.4672e+06, Err 2-norm:  3.4925e+02, inf-norm: 2.9956e+01

image

DRM

python pinn_1d.py   --levels 1 --epochs 20000 --lr 1e-3 --activation tanh --sweeps 1 --plot --hidden_dims 128 128 128 128 --high_freq 16 --bc_weight 500 --nx 2000  --gamma 1 --loss_type 1 --enforce_bc
Iteration    999/ 20000, DRM Loss: -3.6272e+03, Err 2-norm:  4.2620e+01, inf-norm: 1.1399e+01
Iteration   1999/ 20000, DRM Loss: -3.7274e+03, Err 2-norm:  4.2342e+01, inf-norm: 1.1396e+01
Iteration   2999/ 20000, DRM Loss: -3.7761e+03, Err 2-norm:  4.2171e+01, inf-norm: 1.1396e+01
Iteration   3999/ 20000, DRM Loss: -3.7945e+03, Err 2-norm:  4.2251e+01, inf-norm: 1.1396e+01
Iteration   4999/ 20000, DRM Loss: -3.8074e+03, Err 2-norm:  4.2161e+01, inf-norm: 1.1395e+01
Iteration   5999/ 20000, DRM Loss: -3.8168e+03, Err 2-norm:  4.2100e+01, inf-norm: 1.1395e+01
Iteration   6999/ 20000, DRM Loss: -3.8219e+03, Err 2-norm:  4.2085e+01, inf-norm: 1.1395e+01
Iteration   7999/ 20000, DRM Loss: -3.8172e+03, Err 2-norm:  4.2016e+01, inf-norm: 1.1392e+01
Iteration   8999/ 20000, DRM Loss: -3.7533e+03, Err 2-norm:  4.2469e+01, inf-norm: 1.1411e+01
Iteration   9999/ 20000, DRM Loss: -3.8452e+03, Err 2-norm:  4.2098e+01, inf-norm: 1.1396e+01
Iteration  10999/ 20000, DRM Loss: -3.8509e+03, Err 2-norm:  4.2057e+01, inf-norm: 1.1395e+01
Iteration  11999/ 20000, DRM Loss: -3.8582e+03, Err 2-norm:  4.2054e+01, inf-norm: 1.1394e+01
Iteration  12999/ 20000, DRM Loss: -3.8515e+03, Err 2-norm:  4.2009e+01, inf-norm: 1.1393e+01
Iteration  13999/ 20000, DRM Loss: -3.8632e+03, Err 2-norm:  4.2054e+01, inf-norm: 1.1393e+01
Iteration  14999/ 20000, DRM Loss: -3.9252e+03, Err 2-norm:  4.1995e+01, inf-norm: 1.1386e+01
Iteration  15999/ 20000, DRM Loss: -3.8522e+03, Err 2-norm:  4.2245e+01, inf-norm: 1.1404e+01
Iteration  16999/ 20000, DRM Loss: -3.8638e+03, Err 2-norm:  4.2194e+01, inf-norm: 1.1397e+01
Iteration  17999/ 20000, DRM Loss: -3.8744e+03, Err 2-norm:  4.2103e+01, inf-norm: 1.1394e+01
Iteration  18999/ 20000, DRM Loss: -3.8717e+03, Err 2-norm:  4.2069e+01, inf-norm: 1.1395e+01
Iteration  19999/ 20000, DRM Loss: -3.8648e+03, Err 2-norm:  4.2080e+01, inf-norm: 1.1396e+01

image

Dual Cone

python pinn_1d.py   --levels 1 --epochs 20000 --lr 1e-4 --activation tanh --sweeps 1 --plot --hidden_dims 128 128 128 128 --high_freq 16 --bc_weight 500 --nx 2000  --gamma 1 --loss_type 2 --enforce_bc
Iteration    999/ 20000, DCGD(PINN+DRM) Loss: 9.5885e+06, Err 2-norm:  8.6899e+01, inf-norm: 1.1051e+01
Iteration   1999/ 20000, DCGD(PINN+DRM) Loss: 8.6901e+06, Err 2-norm:  1.1592e+02, inf-norm: 1.0718e+01
Iteration   2999/ 20000, DCGD(PINN+DRM) Loss: 8.0062e+06, Err 2-norm:  1.4201e+02, inf-norm: 1.2087e+01
Iteration   3999/ 20000, DCGD(PINN+DRM) Loss: 7.4509e+06, Err 2-norm:  1.6569e+02, inf-norm: 1.4167e+01
Iteration   4999/ 20000, DCGD(PINN+DRM) Loss: 6.9887e+06, Err 2-norm:  1.8726e+02, inf-norm: 1.6040e+01
Iteration   5999/ 20000, DCGD(PINN+DRM) Loss: 6.5988e+06, Err 2-norm:  2.0692e+02, inf-norm: 1.7736e+01
Iteration   6999/ 20000, DCGD(PINN+DRM) Loss: 6.2673e+06, Err 2-norm:  2.2480e+02, inf-norm: 1.9273e+01
Iteration   7999/ 20000, DCGD(PINN+DRM) Loss: 5.9838e+06, Err 2-norm:  2.4106e+02, inf-norm: 2.0665e+01
Iteration   8999/ 20000, DCGD(PINN+DRM) Loss: 5.7404e+06, Err 2-norm:  2.5580e+02, inf-norm: 2.1925e+01
Iteration   9999/ 20000, DCGD(PINN+DRM) Loss: 5.5305e+06, Err 2-norm:  2.6917e+02, inf-norm: 2.3066e+01
Iteration  10999/ 20000, DCGD(PINN+DRM) Loss: 5.3490e+06, Err 2-norm:  2.8128e+02, inf-norm: 2.4098e+01
Iteration  11999/ 20000, DCGD(PINN+DRM) Loss: 5.1914e+06, Err 2-norm:  2.9223e+02, inf-norm: 2.5041e+01
Iteration  12999/ 20000, DCGD(PINN+DRM) Loss: 5.0543e+06, Err 2-norm:  3.0213e+02, inf-norm: 2.5896e+01
Iteration  13999/ 20000, DCGD(PINN+DRM) Loss: 4.9346e+06, Err 2-norm:  3.1109e+02, inf-norm: 2.6668e+01
Iteration  14999/ 20000, DCGD(PINN+DRM) Loss: 4.8298e+06, Err 2-norm:  3.1917e+02, inf-norm: 2.7365e+01
Iteration  15999/ 20000, DCGD(PINN+DRM) Loss: 4.7378e+06, Err 2-norm:  3.2648e+02, inf-norm: 2.7994e+01
Iteration  16999/ 20000, DCGD(PINN+DRM) Loss: 4.6570e+06, Err 2-norm:  3.3307e+02, inf-norm: 2.8562e+01
Iteration  17999/ 20000, DCGD(PINN+DRM) Loss: 4.5857e+06, Err 2-norm:  3.3903e+02, inf-norm: 2.9075e+01
Iteration  18999/ 20000, DCGD(PINN+DRM) Loss: 4.5228e+06, Err 2-norm:  3.4440e+02, inf-norm: 2.9538e+01
Iteration  19999/ 20000, DCGD(PINN+DRM) Loss: 4.4672e+06, Err 2-norm:  3.4925e+02, inf-norm: 2.9956e+01

image

@stevengogogo
Copy link
Collaborator Author

stevengogogo commented Jun 17, 2025 •
edited
Loading

Exp: Dual cone has intermediate perforamance between PINN and DRM

Dual Cone

python pinn_1d.py   --levels 1 --epochs 20000 --lr 1e-4 --activation gelu --sweeps 1 --plot --hidden_dims 128 128 128 128 128 128 --hi
gh_freq 8 --nx 2000  --gamma 1 --loss_type 2 --enforce_bc

Iteration    999/ 20000, DCGD(PINN+DRM) Loss: 2.8127e+04, Err 2-norm:  2.0452e+01, inf-norm: 1.7408e+00
Iteration   1999/ 20000, DCGD(PINN+DRM) Loss: 2.6857e+04, Err 2-norm:  8.0729e+00, inf-norm: 8.0439e-01
Iteration   2999/ 20000, DCGD(PINN+DRM) Loss: 2.6451e+04, Err 2-norm:  5.7338e+00, inf-norm: 7.2658e-01
Iteration   3999/ 20000, DCGD(PINN+DRM) Loss: 2.6333e+04, Err 2-norm:  5.0116e+00, inf-norm: 6.9810e-01
Iteration   4999/ 20000, DCGD(PINN+DRM) Loss: 2.6312e+04, Err 2-norm:  4.6948e+00, inf-norm: 6.7968e-01
Iteration   5999/ 20000, DCGD(PINN+DRM) Loss: 2.6340e+04, Err 2-norm:  4.4111e+00, inf-norm: 6.4824e-01
Iteration   6999/ 20000, DCGD(PINN+DRM) Loss: 2.6284e+04, Err 2-norm:  4.3407e+00, inf-norm: 6.6255e-01
Iteration   7999/ 20000, DCGD(PINN+DRM) Loss: 2.6270e+04, Err 2-norm:  4.2255e+00, inf-norm: 6.5382e-01
Iteration   8999/ 20000, DCGD(PINN+DRM) Loss: 2.6257e+04, Err 2-norm:  4.1554e+00, inf-norm: 6.4195e-01
Iteration   9999/ 20000, DCGD(PINN+DRM) Loss: 2.6243e+04, Err 2-norm:  4.1571e+00, inf-norm: 6.3141e-01
Iteration  10999/ 20000, DCGD(PINN+DRM) Loss: 2.6231e+04, Err 2-norm:  4.2237e+00, inf-norm: 6.2208e-01
Iteration  11999/ 20000, DCGD(PINN+DRM) Loss: 2.6220e+04, Err 2-norm:  4.3576e+00, inf-norm: 6.1712e-01
Iteration  12999/ 20000, DCGD(PINN+DRM) Loss: 2.6210e+04, Err 2-norm:  4.5237e+00, inf-norm: 6.1114e-01
Iteration  13999/ 20000, DCGD(PINN+DRM) Loss: 2.6200e+04, Err 2-norm:  4.6912e+00, inf-norm: 6.0220e-01
Iteration  14999/ 20000, DCGD(PINN+DRM) Loss: 2.6191e+04, Err 2-norm:  4.8873e+00, inf-norm: 5.9581e-01
Iteration  15999/ 20000, DCGD(PINN+DRM) Loss: 2.6183e+04, Err 2-norm:  5.0869e+00, inf-norm: 5.8968e-01
Iteration  16999/ 20000, DCGD(PINN+DRM) Loss: 2.6177e+04, Err 2-norm:  5.2896e+00, inf-norm: 5.8027e-01
Iteration  17999/ 20000, DCGD(PINN+DRM) Loss: 2.6170e+04, Err 2-norm:  5.4869e+00, inf-norm: 5.7811e-01
Iteration  18999/ 20000, DCGD(PINN+DRM) Loss: 2.6165e+04, Err 2-norm:  5.6714e+00, inf-norm: 5.7472e-01
Iteration  19999/ 20000, DCGD(PINN+DRM) Loss: 2.6160e+04, Err 2-norm:  5.8575e+00, inf-norm: 5.9570e-01

image

PINN

python pinn_1d.py   --levels 1 --epochs 20000 --lr 1e-4 --activation gelu --sweeps 1 --plot --hidden_dims 128 128 128 128 128 128 --high_freq 8 --nx 2000  --gamma 1 --loss_type 0 --enforce_bc

Training Level 0
Level 0: TRAIN
Iteration    999/ 20000, PINN Loss: 4.1865e+03, Err 2-norm:  3.7812e+01, inf-norm: 3.2528e+00
Iteration   1999/ 20000, PINN Loss: 2.7049e+02, Err 2-norm:  7.4351e+00, inf-norm: 6.7886e-01
Iteration   2999/ 20000, PINN Loss: 9.0725e+01, Err 2-norm:  4.6114e-01, inf-norm: 4.8280e-02
Iteration   3999/ 20000, PINN Loss: 4.2691e+01, Err 2-norm:  5.6746e-01, inf-norm: 4.8258e-02
Iteration   4999/ 20000, PINN Loss: 3.0162e+01, Err 2-norm:  3.8562e-01, inf-norm: 3.3606e-02
Iteration   5999/ 20000, PINN Loss: 1.6108e+01, Err 2-norm:  1.9068e-01, inf-norm: 1.6662e-02
Iteration   6999/ 20000, PINN Loss: 1.4033e+01, Err 2-norm:  1.6978e-01, inf-norm: 1.9789e-02
Iteration   7999/ 20000, PINN Loss: 7.2634e+00, Err 2-norm:  8.7759e-02, inf-norm: 7.8963e-03
Iteration   8999/ 20000, PINN Loss: 5.2322e+00, Err 2-norm:  5.7520e-02, inf-norm: 5.0458e-03
Iteration   9999/ 20000, PINN Loss: 3.7976e+00, Err 2-norm:  4.9099e-02, inf-norm: 4.6464e-03
Iteration  10999/ 20000, PINN Loss: 6.5584e+00, Err 2-norm:  7.5448e-02, inf-norm: 9.3102e-03
Iteration  11999/ 20000, PINN Loss: 2.2596e+00, Err 2-norm:  1.2351e-02, inf-norm: 1.4109e-03
Iteration  12999/ 20000, PINN Loss: 1.9509e+00, Err 2-norm:  4.4215e-02, inf-norm: 4.0745e-03
Iteration  13999/ 20000, PINN Loss: 1.6758e+00, Err 2-norm:  2.6122e-02, inf-norm: 2.7473e-03
Iteration  14999/ 20000, PINN Loss: 1.2470e+00, Err 2-norm:  5.9611e-03, inf-norm: 7.9978e-04
Iteration  15999/ 20000, PINN Loss: 1.0546e+00, Err 2-norm:  5.4022e-03, inf-norm: 6.3655e-04
Iteration  16999/ 20000, PINN Loss: 9.0720e-01, Err 2-norm:  9.4335e-03, inf-norm: 1.0395e-03
Iteration  17999/ 20000, PINN Loss: 7.9355e-01, Err 2-norm:  5.1088e-03, inf-norm: 5.0555e-04
Iteration  18999/ 20000, PINN Loss: 6.8692e-01, Err 2-norm:  6.5046e-03, inf-norm: 5.4762e-04
Iteration  19999/ 20000, PINN Loss: 6.4154e-01, Err 2-norm:  1.5080e-02, inf-norm: 1.4453e-03

image

DRM

python pinn_1d.py   --levels 1 --epochs 20000 --lr 1e-4 --activation gelu --sweeps 1 --plot --hidden_dims 128 128 128 128 128 128 --high_freq 8 --nx 2000  --gamma 1 --loss_type 1 --enforce_bc

Iteration    999/ 20000, DRM Loss: -3.2700e+03, Err 2-norm:  8.3493e+01, inf-norm: 8.0330e+00
Iteration   1999/ 20000, DRM Loss: -3.4581e+03, Err 2-norm:  8.4489e+01, inf-norm: 8.1039e+00
Iteration   2999/ 20000, DRM Loss: -3.4839e+03, Err 2-norm:  8.4612e+01, inf-norm: 8.0925e+00
Iteration   3999/ 20000, DRM Loss: -3.4844e+03, Err 2-norm:  8.4583e+01, inf-norm: 8.0900e+00
Iteration   4999/ 20000, DRM Loss: -3.4844e+03, Err 2-norm:  8.4580e+01, inf-norm: 8.0899e+00
Iteration   5999/ 20000, DRM Loss: -3.4844e+03, Err 2-norm:  8.4578e+01, inf-norm: 8.0899e+00
Iteration   6999/ 20000, DRM Loss: -3.4844e+03, Err 2-norm:  8.4578e+01, inf-norm: 8.0900e+00
Iteration   7999/ 20000, DRM Loss: -3.4844e+03, Err 2-norm:  8.4579e+01, inf-norm: 8.0900e+00
Iteration   8999/ 20000, DRM Loss: -3.4843e+03, Err 2-norm:  8.4731e+01, inf-norm: 8.0972e+00
Iteration   9999/ 20000, DRM Loss: -3.4844e+03, Err 2-norm:  8.4580e+01, inf-norm: 8.0901e+00
Iteration  10999/ 20000, DRM Loss: -3.4845e+03, Err 2-norm:  8.4581e+01, inf-norm: 8.0902e+00
Iteration  11999/ 20000, DRM Loss: -3.4845e+03, Err 2-norm:  8.4578e+01, inf-norm: 8.0900e+00
Iteration  12999/ 20000, DRM Loss: -3.4845e+03, Err 2-norm:  8.4578e+01, inf-norm: 8.0901e+00
Iteration  13999/ 20000, DRM Loss: -3.4845e+03, Err 2-norm:  8.4579e+01, inf-norm: 8.0901e+00
Iteration  14999/ 20000, DRM Loss: -3.4845e+03, Err 2-norm:  8.4579e+01, inf-norm: 8.0901e+00
Iteration  15999/ 20000, DRM Loss: -3.4845e+03, Err 2-norm:  8.4579e+01, inf-norm: 8.0901e+00
Iteration  16999/ 20000, DRM Loss: -3.4845e+03, Err 2-norm:  8.4572e+01, inf-norm: 8.0897e+00
Iteration  17999/ 20000, DRM Loss: -3.4845e+03, Err 2-norm:  8.4579e+01, inf-norm: 8.0901e+00
Iteration  18999/ 20000, DRM Loss: -3.4844e+03, Err 2-norm:  8.4473e+01, inf-norm: 8.0850e+00
Iteration  19999/ 20000, DRM Loss: -3.4845e+03, Err 2-norm:  8.4579e+01, inf-norm: 8.0900e+00

image

@stevengogogo
Copy link
Collaborator Author

stevengogogo commented Jun 17, 2025 •
edited
Loading

Idea: Gradient belief to help getting out local minima via mutual global minima

Both PINN loss and DRM loss lead to the same solution. The major task we want is to avoid local minima. Thus, we should trust the large gradient than the small one if one of loss in local minima.

Does PINN loss and DRM loss has same local mimia? We only know they have same global minima

Strategy:
grad1 and grad2 can be applied to different optimizer

  1. Max(norm(grad1), norm(grad2))
  2. Record grad1 and grad2 to check if they are in local minima. Belief the one that is not. This minimize the noise in SGD

@stevengogogo
Copy link
Collaborator Author

I think we can keep this as draft. Since this is duplicate with #26 but with original implementation of Dual Cone PINN paper

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
experiment
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@stevengogogo