Skip to content

Commit 80ce729

Browse files
mfazrinizarkozistr
mfazrinizar
and
kozistr
authored
[Feature] Implement LoRARite Optimizer (#504)
* feat: implement LoRARite optimizer * refactor: update pytorch_optimizer/optimizer/lora_rite.py, removing extra bool exclusion Co-authored-by: Hyeongchan Kim <kozistr@gmail.com> * test: move LoRARite optimizer test to test_optimizers.py * refactor: remove runtime bool checks, pair_infos storage, and grad norm device is lazily initialized * fix: resolve optimizer merge cleanup * fix: resolve LoRARite CI cleanup --------- Co-authored-by: Hyeongchan Kim <kozistr@gmail.com>
1 parent 216c7a8 commit 80ce729

10 files changed

Lines changed: 619 additions & 2 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -201,6 +201,7 @@ get_supported_optimizers(['adam*', 'ranger*'])
201201
| PAdam | *Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks* | [github](https://github.com/uclaml/Padam) | [paper](https://arxiv.org/abs/1806.06763)([cite](https://github.com/uclaml/Padam#citation)) |
202202
| LOMO | *Full Parameter Fine-tuning for Large Language Models with Limited Resources* | [github](https://github.com/OpenLMLab/LOMO) | [paper](https://arxiv.org/abs/2306.09782)([cite](https://github.com/OpenLMLab/LOMO#citation)) |
203203
| AdaLOMO | *Low-memory Optimization with Adaptive Learning Rate* | [github](https://github.com/OpenLMLab/LOMO) | [paper](https://arxiv.org/abs/2310.10195)([cite](https://github.com/OpenLMLab/LOMO#citation)) |
204+
| LoRARite | *LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization* | [github](https://github.com/gkevinyen5418/LoRA-RITE) | [paper](https://arxiv.org/abs/2410.20625)([cite](https://ui.adsabs.harvard.edu/abs/2024arXiv241020625Y/exportcitation)) |
204205
| Tiger | *A Tight-fisted Optimizer, an optimizer that is extremely budget-conscious* | [github](https://github.com/bojone/tiger) | [cite](https://github.com/bojone/tiger/blob/main/README_en.md#citation) |
205206
| CAME | *Confidence-guided Adaptive Memory Efficient Optimization* | [github](https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/CAME) | [paper](https://aclanthology.org/2023.acl-long.243/)([cite](https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/CAME#citation)) |
206207
| WSAM | *Sharpness-Aware Minimization Revisited: Weighted Sharpness as a Regularization Term* | [github](https://github.com/intelligent-machine-learning/dlrover/blob/master/atorch/atorch/optimizers/wsam.py) | [paper](https://arxiv.org/abs/2305.15817)([cite](https://github.com/intelligent-machine-learning/dlrover)) |

‎docs/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -201,6 +201,7 @@ get_supported_optimizers(['adam*', 'ranger*'])
201201
| PAdam | *Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks* | [github](https://github.com/uclaml/Padam) | [paper](https://arxiv.org/abs/1806.06763)([cite](https://github.com/uclaml/Padam#citation)) |
202202
| LOMO | *Full Parameter Fine-tuning for Large Language Models with Limited Resources* | [github](https://github.com/OpenLMLab/LOMO) | [paper](https://arxiv.org/abs/2306.09782)([cite](https://github.com/OpenLMLab/LOMO#citation)) |
203203
| AdaLOMO | *Low-memory Optimization with Adaptive Learning Rate* | [github](https://github.com/OpenLMLab/LOMO) | [paper](https://arxiv.org/abs/2310.10195)([cite](https://github.com/OpenLMLab/LOMO#citation)) |
204+
| LoRARite | *LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization* | [github](https://github.com/gkevinyen5418/LoRA-RITE) | [paper](https://arxiv.org/abs/2410.20625)([cite](https://ui.adsabs.harvard.edu/abs/2024arXiv241020625Y/exportcitation)) |
204205
| Tiger | *A Tight-fisted Optimizer, an optimizer that is extremely budget-conscious* | [github](https://github.com/bojone/tiger) | [cite](https://github.com/bojone/tiger/blob/main/README_en.md#citation) |
205206
| CAME | *Confidence-guided Adaptive Memory Efficient Optimization* | [github](https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/CAME) | [paper](https://aclanthology.org/2023.acl-long.243/)([cite](https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/CAME#citation)) |
206207
| WSAM | *Sharpness-Aware Minimization Revisited: Weighted Sharpness as a Regularization Term* | [github](https://github.com/intelligent-machine-learning/dlrover/blob/master/atorch/atorch/optimizers/wsam.py) | [paper](https://arxiv.org/abs/2305.15817)([cite](https://github.com/intelligent-machine-learning/dlrover)) |

‎docs/optimizer.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -348,6 +348,10 @@
348348
:docstring:
349349
:members:
350350

351+
::: pytorch_optimizer.LoRARite
352+
:docstring:
353+
:members:
354+
351355
::: pytorch_optimizer.MADGRAD
352356
:docstring:
353357
:members:

‎pyproject.toml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@ keywords = [
1515
"AdaSmooth", "AdEMAMix", "Simplified-AdEMAMix", "ADOPT", "AdaHessian", "Adai", "Adalite", "AdaLomo", "AdamMini",
1616
"AdamP", "AdamS", "Adan", "AggMo", "Aida", "AliG", "Amos", "Apollo", "APOLLO", "AvaGrad", "bSAM", "BCOS", "CAME",
1717
"Conda", "DAdaptAdaGrad", "DAdaptAdam", "DAdaptAdan", "DAdaptSGD", "DAdaptLion", "DeMo", "DiffGrad", "EmoFact",
18+
"EmoLynx", "EmoNavi", "EmoNeco", "EmoZeal", "EXAdam", "FAdam", "Fira", "FOCUS", "FriendlySAM", "Fromage", "FTRL",
19+
"GaLore", "Grams", "Gravity", "GrokFast", "GSAM", "Kate", "Lamb", "LaProp", "LARS", "Lion", "LoRARite", "LOMO", "Lookahead",
1820
"EmoLynx", "EmoNavi", "EmoNeco", "EmoZeal", "EXAdam", "FAdam", "Fira", "FlashAdamW", "FOCUS", "FriendlySAM",
1921
"Conda", "DAdaptAdaGrad", "DAdaptAdam", "DAdaptAdan", "DAdaptSGD", "DAdaptLion", "DeMo", "DiffGrad", "DualAdam",
2022
"EmoFact", "EmoLynx", "EmoNavi", "EmoNeco", "EmoZeal", "EXAdam", "FAdam", "Fira", "FOCUS", "FriendlySAM", "Fromage",

‎pytorch_optimizer/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -143,6 +143,7 @@
143143
Lamb,
144144
LaProp,
145145
Lion,
146+
LoRARite,
146147
Lookahead,
147148
LookSAM,
148149
Muon,

‎pytorch_optimizer/optimizer/__init__.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@
6969
from pytorch_optimizer.optimizer.lion import Lion
7070
from pytorch_optimizer.optimizer.lomo import LOMO, AdaLOMO
7171
from pytorch_optimizer.optimizer.lookahead import Lookahead
72+
from pytorch_optimizer.optimizer.lora_rite import LoRARite
7273
from pytorch_optimizer.optimizer.madgrad import MADGRAD
7374
from pytorch_optimizer.optimizer.mars import MARS
7475
from pytorch_optimizer.optimizer.msvag import MSVAG
@@ -195,6 +196,7 @@
195196
Kron,
196197
LARS,
197198
LOMO,
199+
LoRARite,
198200
LaProp,
199201
Lamb,
200202
Lion,

0 commit comments

Comments
 (0)