Search results

Stability concerns when applying cuda-checkpoint --launch-job to existing training workloads

Status: Open.

#55 In NVIDIA/cuda-checkpoint;

· amazingguni opened on Jul 2, 2026

Killing a cuda-checkpoint restored process hard-hangs the whole host (bare metal, open 595.71.05) — suspending first avoids it

Status: Open.

#53 In NVIDIA/cuda-checkpoint;

· imlach opened on Jun 11, 2026

What is the correct procedure to successfully dump a GPU workload (SGLang) running inside an NVIDIA container with cuda-checkpoint + CRIU?

Status: Open.

#52 In NVIDIA/cuda-checkpoint;

· wangzhen2271 opened on Jun 11, 2026

Feature request: GDS (cuFile) or general RDMA transport for checkpoint/restore

Status: Open.

#49 In NVIDIA/cuda-checkpoint;

· 4D0R opened on May 14, 2026

cuda-checkpoint default restore preserves stale cuDeviceGetPCIBusId cache across cross-node gpu migration; NVML lookups fail as NVML look ups include cudaDeviceGetPciBusId() which returns old id and causes failure post restore

Status: Open.

#48 In NVIDIA/cuda-checkpoint;

· shubh3794 opened on Apr 22, 2026

Could not restore on process ID 3406143: "invalid argument"

Status: Open.

#47 In NVIDIA/cuda-checkpoint;

· wangzhen2271 opened on Apr 16, 2026

Is it not possible to restore from GPU 0 to GPU 0?

Status: Open.

#46 In NVIDIA/cuda-checkpoint;

· wangzhen2271 opened on Apr 9, 2026

cuda-checkpoint --toggle hangs when checkpointing one process of torch.distributed NCCL job

Status: Open.

#45 In NVIDIA/cuda-checkpoint;

· Taya666 opened on Feb 10, 2026

Restore permanently fails after initial OOM — process stuck in CHECKPOINTED state

Status: Open.

#44 In NVIDIA/cuda-checkpoint;

· asayyah opened on Jan 28, 2026

590 Features

Status: Open.

#43 In NVIDIA/cuda-checkpoint;

· rst0git opened on Dec 18, 2025

Fails with "out of memory" on MIG-enabled GPUs

Status: Open.

#42 In NVIDIA/cuda-checkpoint;

· ayushr2 opened on Dec 17, 2025

cuda-checkpoint fails to throw error when performing CRIU on non-CUDA processes

Status: Open.

#41 In NVIDIA/cuda-checkpoint;

· xuhongqiang-kernel opened on Dec 16, 2025