Update torchft.md #1596
Update torchft.md #1596
Conversation
H-Huang
requested review from
tianyu-l,
fegin,
wwwjn and
wconstab
as code owners
meta-cla
bot
added
the
CLA Signed
There was a problem hiding this comment.
(adding a small comment if you don't mind as I have been using diloco with torchtitan for a bit)
could be usefull to add a clarification that the global_batch_size is split across all the diloco island. It's clear for HSDP but for diloco there is a distinction between global batch size (all diloco island) and the inner global batch size (inside each diloco island doing normal sharding)
H-Huang
force-pushed
the
torchft_readme
branch
from
c87fd44 to
d65112f
Compare
|
@samsja thanks :), added |
There was a problem hiding this comment.
what's the relationship with https://github.com/pytorch/torchtitan/blob/main/docs/torchft.md
Could we consolidate to just keep one of them?
|
@tianyu-l, oops I didn't realize. Vishal had asked if there were any docs and i didn't realize there was an existing one, so i just added something new. Let me see if there is anything to consolidate. If this is redundant, then ill just close this PR |
H-Huang
force-pushed
the
torchft_readme
branch
from
26113b0 to
d86e252
Compare
| - `--fault_tolerance.sync_steps`: The number of training steps before synchronization. | ||
| - `--fault_tolerance.semi_sync_method`: Synchronization method (e.g., "local_sgd", "diloco") | ||
|
|
||
| For more semi-synchronouse configuration options, see [ft/config/job_config.py](config/job_config.py). |
There was a problem hiding this comment.
Probably just run NGPU=1 ./run_train.sh --help to get the help message instead of understanding job_config.py.
Add some basic documentation on how to use Titan with TorchFt for DiLoCo.
LMK if anything needs clarification @vishal9-team