Improve GPU-aware section in the docs by luraess · Pull Request #927 · JuliaParallel/MPI.jl

luraess · 2025-12-17T09:22:54Z

Adds infos to the doc as per discussion in #924.

docs/src/usage.md

aklocker42 · 2025-12-17T09:39:18Z

docs/src/usage.md

+> comm_loc = MPI.Comm_split_type(comm, MPI.COMM_TYPE_SHARED, rank)
+> rank_loc = MPI.Comm_rank(comm_loc)
+> ```
+> If using (2), one can use the default device but make sur to handle device visbility in the scheduler; for SLURM on Cray systems, this can be mostly achieved using `--gpus-per-task=1`.


Doesn't '--gpus-per-task' for SLURM prevent the use of GPU Peer2Peer IPC mechanisms (https://cpe.ext.hpe.com/docs/24.03/mpt/mpich/intro_mpi.html) which would have a negative impact on performance?

Yeah that's what I also remember, but perhaps Nvidia has finally fixed this?

Nope, not as far as I can tell.

Thanks for pointing this out. I can make the text more generic

docs/src/usage.md

vchuravy · 2025-12-17T19:24:52Z

docs/src/usage.md

+Successfully running the [alltoall\_test\_cuda.jl](https://gist.github.com/luraess/0063e90cb08eb2208b7fe204bbd90ed2)
+should confirm your MPI implementation to have the CUDA support enabled. Moreover, successfully running the
+[alltoall\_test\_cuda\_multigpu.jl](https://gist.github.com/luraess/ed93cc09ba04fe16f63b4219c1811566) should confirm


Can we move the files into this repository?

Yes, we can

Were shall one put them?

vchuravy · 2025-12-17T19:25:18Z

docs/src/usage.md

+!!! note "Preloads"
+    On Cray machines, you may need to ensure the following preloads to be set in the preferences:
+    ```
+    preloads = ["libmpi_gtl_hsa.so"]
+    preloads_env_switch = "MPICH_GPU_SUPPORT_ENABLED"
+    ```


This is also true for CUDA.

vchuravy · 2025-12-17T19:26:21Z

docs/src/usage.md

+    preloads_env_switch = "MPICH_GPU_SUPPORT_ENABLED"
+    ```
+
+!!! note "Multiple GPUs per node"


Suggested change

!!! note "Multiple GPUs per node"

### "Multiple GPUs per node"

Since the text is not just on ROCM?

luraess · 2026-01-06T09:23:03Z

Happy if someone would review added file naming and location, and updated text 🙏

lcw · 2026-01-13T21:21:42Z

This looks good to me. It seems all issues were addressed. Thanks!

giordano · 2026-01-26T11:52:06Z

This PR broke building the docs

┌ Error: invalid local link/image: path pointing to a file outside of build directory in docs/src/usage.md
│   link =
│    @ast MarkdownAST.Link("../examples/alltoall_test_cuda.jl", "") do
│      MarkdownAST.Text("alltoall")
│      MarkdownAST.Text("_")
│      MarkdownAST.Text("test")
│      MarkdownAST.Text("_")
│      MarkdownAST.Text("cuda.jl")
│    end
│    
└ @ Documenter ~/.julia/packages/Documenter/xvqbW/src/utilities/utilities.jl:47
┌ Error: invalid local link/image: path pointing to a file outside of build directory in docs/src/usage.md
│   link =
│    @ast MarkdownAST.Link("../examples/alltoall_test_cuda_multigpu.jl", "") do
│      MarkdownAST.Text("alltoall")
│      MarkdownAST.Text("_")
│      MarkdownAST.Text("test")
│      MarkdownAST.Text("_")
│      MarkdownAST.Text("cuda")
│      MarkdownAST.Text("_")
│      MarkdownAST.Text("multigpu.jl")
│    end
│    
└ @ Documenter ~/.julia/packages/Documenter/xvqbW/src/utilities/utilities.jl:47
┌ Error: invalid local link/image: path pointing to a file outside of build directory in docs/src/usage.md
│   link =
│    @ast MarkdownAST.Link("../examples/alltoall_test_rocm.jl", "") do
│      MarkdownAST.Text("alltoall")
│      MarkdownAST.Text("_")
│      MarkdownAST.Text("test")
│      MarkdownAST.Text("_")
│      MarkdownAST.Text("rocm.jl")
│    end
│    
└ @ Documenter ~/.julia/packages/Documenter/xvqbW/src/utilities/utilities.jl:47
┌ Error: invalid local link/image: path pointing to a file outside of build directory in docs/src/usage.md
│   link =
│    @ast MarkdownAST.Link("../examples/alltoall_test_rocm_multigpu.jl", "") do
│      MarkdownAST.Text("alltoall")
│      MarkdownAST.Text("_")
│      MarkdownAST.Text("test")
│      MarkdownAST.Text("_")
│      MarkdownAST.Text("rocm")
│      MarkdownAST.Text("_")
│      MarkdownAST.Text("multigpu.jl")
│    end
│    
└ @ Documenter ~/.julia/packages/Documenter/xvqbW/src/utilities/utilities.jl:47
[ Info: CheckDocument: running document checks.

luraess · 2026-01-26T14:27:34Z

Yeah, this was somehow my open question in "review added file naming and location", as I would suspect it may not work like other examples but was unsure whether to put those examples in another folder. Any suggestions @giordano ?

Improve GPU-aware section

61e0ede

luraess changed the title ~~Improve GPU-aware section~~ Improve GPU-aware section in the docs Dec 17, 2025

luraess added the documentation label Dec 17, 2025

luraess commented Dec 17, 2025

View reviewed changes

docs/src/usage.md Outdated Show resolved Hide resolved

luraess mentioned this pull request Dec 17, 2025

issues with test of CUDA-aware MPI support #924

Closed

aklocker42 reviewed Dec 17, 2025

View reviewed changes

giordano reviewed Dec 17, 2025

View reviewed changes

docs/src/usage.md Outdated Show resolved Hide resolved

Add suggestions

b8312ea

vchuravy reviewed Dec 17, 2025

View reviewed changes

luraess added 2 commits December 17, 2025 21:05

Update

a017080

Add examples

d64e598

Fix spelling typo.

55f2fe9

lcw merged commit 7dabd91 into master Jan 13, 2026
1 of 4 checks passed

lcw deleted the lr/update-doc branch January 13, 2026 21:23

This was referenced Mar 11, 2026

Implement new ABI "MPIABI" #925

Merged

GPU-aware example break building docs #936

Closed

	!!! note "Multiple GPUs per node"
	### "Multiple GPUs per node"

Uh oh!

Conversation

luraess commented Dec 17, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luraess commented Jan 6, 2026

Uh oh!

lcw commented Jan 13, 2026

Uh oh!

Uh oh!

giordano commented Jan 26, 2026

Uh oh!

luraess commented Jan 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants