We are integrating cuda-checkpoint into our training platform by wrapping existing jobs with cuda-checkpoint --launch-job.
We simply prepend the launcher so that checkpoint/restore becomes available on demand.
Before deploying this broadly, we'd like to understand how much attaching --launch-job affects the underlying workload
if we transparently attach cuda-checkpoint --launch-job to every training job, how confident can we be that jobs behave identically to running them directly?
Any data, known limitations, or recommendations from the team would be very helpful. Thanks!
We are integrating cuda-checkpoint into our training platform by wrapping existing jobs with
cuda-checkpoint --launch-job.We simply prepend the launcher so that checkpoint/restore becomes available on demand.
Before deploying this broadly, we'd like to understand how much attaching
--launch-jobaffects the underlying workloadif we transparently attach
cuda-checkpoint --launch-jobto every training job, how confident can we be that jobs behave identically to running them directly?Any data, known limitations, or recommendations from the team would be very helpful. Thanks!