Question about the multimem in NCCL

[jiangaojie]

Hello, authors. Thank you for your wonderful work!
I have a question about multimem in H100 GPU. In Figure 5 ( Compare all-reduce(default) with all-reduce(multimem) ), pytorch.distribute.all_reduce() is considered as the result of turning off multimem , so is NCCL's Multimem function (NVLS) turned off by default in VLLM?

[jiangaojie]

Do I need to set the environment variable NCCL_NVLS_ENABLE=True/False to reproduce the results in Figure 5?

[rajagond]

Hi,

Thank you for your interest!

vLLM mostly uses pytorch.distributed for communication collectives such as all_reduce. To use Multimem capabilities on H100 GPUs, we need to perform memory registration. Recently, PyTorch introduced symmetric memory, which provides a cleaner interface for allocating buffers and obtaining Multimem handles. For more details, please refer to this post: PyTorch SymmetricMemory: Harnessing NVLink Programmability with Ease.

Currently, vLLM doesn't support this feature. A pull request is in progress — see the related PR here: vllm-project/vllm#20759.

We used torch==2.6.0 with default parameters to generate Figure 5. As far as we know, pytorch.distributed.all_reduce does not support Multimem-based AllReduce in torch==2.6.0. See the discussion here for more details: microsoft/mscclpp#477 (comment).

If we use torch.empty without NVLS memory registration, we would need to:

1. Copy the original data into a temporary Multimem buffer,
2. Perform the Multimem AllReduce,
3. Copy the result back into the original tensor.

This approach would introduce additional overhead and be inefficient.

[jiangaojie]

Thank you for your reply！

Your test program torch-ncc-allreduce seems to indicate that PyTorch also supports NVLS (multimem-baed all-reduce), is that correct?

But when I am testing the performance impact of NVLS using vLLM on 8 H100 GPU system, whether I enabled NCCL_NVLS_ENABLE or not, there seemed to be no difference in performance . I am confused about this.

Thanks!

[rajagond]

I think if pytorch.distributed doesn't support it, then calling pytorch.distributed.all_reduce with or without the NCCL_NVLS_ENABLE environment variable won't make any difference. In torch==2.6.0, it wasn't supported, see: https://discuss.pytorch.org/t/nvls-support-in-pytorch/217616. I'm not sure about the current PyTorch version.

If you want to compare with and without NCCL_NVLS_ENABLE, I think it's better to use NVIDIA/nccl-tests.

[jiangaojie]

Thanks. It seems released pytorch dosen't support NVLS until now.