Feature request: GDS (cuFile) or general RDMA transport for checkpoint/restore
Right now cuda-checkpoint stages all device memory through host DRAM. On restore that's an extra hop on the read path: bytes have to land in host memory before they can be DMA'd back to the GPU. On checkpoint it stretches out time-to-durability, since the write can't be acknowledged by storage until it's already been copied into host RAM. With a GDS-capable filesystem sitting right there, skipping the bounce would help both sides.
I'd love to know if either of these is on the roadmap:
- Native GDS / cuFile integration. A first-class cuFile path for
checkpoint and restore so the driver can talk to GDS-capable storage
directly.
- A broader RDMA-capable API. Something general enough to back
RDMA-to-storage or other use cases, not tied to a single transport.
There's already a fair bit of related discussion.
A few questions:
- Is GDS / cuFile output planned?
- Is RDMA still backlog as of 590, or has the design moved?
- Anything structural making either of these particularly difficult?
Given some discussions around fast start ups for vllm, I think that a GDS or RDMA based option would be really interesting here. This could help further reduce inference cold starts, and cold starts for any number of other jobs.
Feature request: GDS (cuFile) or general RDMA transport for checkpoint/restore
Right now
cuda-checkpointstages all device memory through host DRAM. On restore that's an extra hop on the read path: bytes have to land in host memory before they can be DMA'd back to the GPU. On checkpoint it stretches out time-to-durability, since the write can't be acknowledged by storage until it's already been copied into host RAM. With a GDS-capable filesystem sitting right there, skipping the bounce would help both sides.I'd love to know if either of these is on the roadmap:
checkpoint and restore so the driver can talk to GDS-capable storage
directly.
RDMA-to-storage or other use cases, not tied to a single transport.
There's already a fair bit of related discussion.
CU_CHECKPOINT_DIRECT_IOsketch in the comments).
A few questions:
Given some discussions around fast start ups for vllm, I think that a GDS or RDMA based option would be really interesting here. This could help further reduce inference cold starts, and cold starts for any number of other jobs.