What is the correct procedure to successfully dump a GPU workload (SGLang) running inside an NVIDIA container with cuda-checkpoint + CRIU?

[wangzhen2271]

I start an NVIDIA container, then launch an SGLang service inside it.
First I run the CUDA checkpoint pre-freeze command on the target process:

/home/wz/cuda-checkpoint/bin/x86_64_Linux/cuda-checkpoint --toggle --pid 2920058

After that, I execute CRIU dump on the host for the same PID:

sudo /home/wz/criu-host/criu/criu dump --shell-job --images-dir /home/wz/ --tree 2919696

This throws the mount sharing error:

Error (criu/mount.c:1092): mnt: Mount 7894 ./proc/driver/nvidia/gpus/0000:ab:00.0 (master_id: 26 shared_id: 0) has unreachable sharing. Try --enable-external-masters.

I added --enable-external-masters --external mnt[] to the CRIU dump command as suggested:

sudo /home/wz/criu-host/criu/criu dump --shell-job --images-dir /home/wz/ --tree 2919696 --enable-external-masters --external mnt[]

Now the dump fails with mapping/plugin errors:

Error (criu/proc_parse.c:118): handle_device_vma plugin failed: No such file or directory
Error (criu/proc_parse.c:672): Can't handle non-regular mapping on 2919696's map 7fc08d672000
Error (criu/cr-dump.c:1589): Collect mappings (pid: 2919696) failed with -1
Error (criu/cr-dump.c:2130): Dumping FAILED.

So what is the correct procedure to successfully dump a GPU workload (SGLang) running inside an NVIDIA container with cuda-checkpoint + CRIU?

[rst0git]

@wangzhen2271 Please feel free to ask any follow up questions in checkpoint-restore/criu#3040