failing `GpuUtilsMpiTests` test for GROMACS 2026.2 due to UCX/Infiniband/RDMA error

[boegel]

follow-up for failing test reports from HPC-UGent GPU clusters accelgor (A100) and litleo (H100), see:

• {bio}[foss/2025b] GROMACS v2026.2 w/ CUDA 12.9.1 #26273

 39/103 Test  #39: GpuUtilsMpiTests .............................***Failed    6.36 sec
�[0;32m[==========] �[mRunning 15 tests from 1 test suite.
�[0;32m[----------] �[mGlobal test environment set-up.
�[0;32m[----------] �[m15 tests from WorksOnParameters/GpuAwareMpiTest
�[0;32m[ RUN      ] �[mWorksOnParameters/GpuAwareMpiTest.SendFromUnpinnedHostBufferToDeviceBuffer/0_values_in_message
�[0;32m[       OK ] �[mWorksOnParameters/GpuAwareMpiTest.SendFromUnpinnedHostBufferToDeviceBuffer/0_values_in_message (0 ms)
�[0;32m[ RUN      ] �[mWorksOnParameters/GpuAwareMpiTest.SendFromUnpinnedHostBufferToDeviceBuffer/1_values_in_message
�[0;32m[       OK ] �[mWorksOnParameters/GpuAwareMpiTest.SendFromUnpinnedHostBufferToDeviceBuffer/1_values_in_message (0 ms)
�[0;32m[ RUN      ] �[mWorksOnParameters/GpuAwareMpiTest.SendFromUnpinnedHostBufferToDeviceBuffer/12000_values_in_message
�[0;32m[       OK ] �[mWorksOnParameters/GpuAwareMpiTest.SendFromUnpinnedHostBufferToDeviceBuffer/12000_values_in_message (0 ms)
�[0;32m[ RUN      ] �[mWorksOnParameters/GpuAwareMpiTest.SendFromPinnedHostBufferToDeviceBuffer/0_values_in_message
�[0;32m[       OK ] �[mWorksOnParameters/GpuAwareMpiTest.SendFromPinnedHostBufferToDeviceBuffer/0_values_in_message (0 ms)
�[0;32m[ RUN      ] �[mWorksOnParameters/GpuAwareMpiTest.SendFromPinnedHostBufferToDeviceBuffer/1_values_in_message
�[0;32m[       OK ] �[mWorksOnParameters/GpuAwareMpiTest.SendFromPinnedHostBufferToDeviceBuffer/1_values_in_message (1 ms)
�[0;32m[ RUN      ] �[mWorksOnParameters/GpuAwareMpiTest.SendFromPinnedHostBufferToDeviceBuffer/12000_values_in_message
�[0;32m[       OK ] �[mWorksOnParameters/GpuAwareMpiTest.SendFromPinnedHostBufferToDeviceBuffer/12000_values_in_message (1 ms)
�[0;32m[ RUN      ] �[mWorksOnParameters/GpuAwareMpiTest.SendFromDeviceBufferToUnpinnedHostBuffer/0_values_in_message
�[0;32m[       OK ] �[mWorksOnParameters/GpuAwareMpiTest.SendFromDeviceBufferToUnpinnedHostBuffer/0_values_in_message (0 ms)
�[0;32m[ RUN      ] �[mWorksOnParameters/GpuAwareMpiTest.SendFromDeviceBufferToUnpinnedHostBuffer/1_values_in_message
�[0;32m[       OK ] �[mWorksOnParameters/GpuAwareMpiTest.SendFromDeviceBufferToUnpinnedHostBuffer/1_values_in_message (3 ms)
�[0;32m[ RUN      ] �[mWorksOnParameters/GpuAwareMpiTest.SendFromDeviceBufferToUnpinnedHostBuffer/12000_values_in_message
[node3904:3487992:0:3487992] ib_mlx5_log.c:179  Remote operation error on mlx5_0:1/IB (synd 0x14 vend 0x89 hw_synd 0/0)
[node3904:3487992:0:3487992] ib_mlx5_log.c:179  RC QP 0x2230 wqe[0]: RDMA_READ s-- [rva 0x14ed43200000 rkey 0xc1fe] [va 0x3579590 len 48000 lkey 0xd700] [rqpn 0x2231 dlid=556 sl=0 port=1 src_path_bits=0]
==== backtrace (tid:3487992) ====
 0 0x00000000000149ce uct_ib_mlx5_completion_with_err()  /tmp/vsc40023/easybuild_build/UCX/1.19.0/GCCcore-14.3.0/ucx-1.19.0/src/uct/ib/mlx5/ib_mlx5_log.c:179
 1 0x0000000000028a8c uct_rc_mlx5_iface_handle_failure()  /tmp/vsc40023/easybuild_build/UCX/1.19.0/GCCcore-14.3.0/ucx-1.19.0/src/uct/ib/mlx5/rc/rc_mlx5_iface.c:235
 2 0x0000000000028a8c uct_rc_iface_arbiter_dispatch()  /tmp/vsc40023/easybuild_build/UCX/1.19.0/GCCcore-14.3.0/ucx-1.19.0/src/uct/ib/rc/base/rc_iface.h:455
 3 0x0000000000028a8c uct_rc_mlx5_iface_handle_failure()  /tmp/vsc40023/easybuild_build/UCX/1.19.0/GCCcore-14.3.0/ucx-1.19.0/src/uct/ib/mlx5/rc/rc_mlx5_iface.c:238
 4 0x0000000000015a85 uct_ib_mlx5_check_completion()  /tmp/vsc40023/easybuild_build/UCX/1.19.0/GCCcore-14.3.0/ucx-1.19.0/src/uct/ib/mlx5/ib_mlx5.c:477
 5 0x000000000002aeb7 uct_ib_mlx5_poll_cq()  /tmp/vsc40023/easybuild_build/UCX/1.19.0/GCCcore-14.3.0/ucx-1.19.0/src/uct/ib/mlx5/ib_mlx5.inl:178
 6 0x000000000002aeb7 uct_rc_mlx5_iface_poll_tx()  /tmp/vsc40023/easybuild_build/UCX/1.19.0/GCCcore-14.3.0/ucx-1.19.0/src/uct/ib/mlx5/rc/rc_mlx5.inl:1891
 7 0x000000000002aeb7 uct_rc_mlx5_iface_progress()  /tmp/vsc40023/easybuild_build/UCX/1.19.0/GCCcore-14.3.0/ucx-1.19.0/src/uct/ib/mlx5/rc/rc_mlx5_iface.c:127
 8 0x000000000002aeb7 uct_rc_mlx5_iface_progress_cyclic()  /tmp/vsc40023/easybuild_build/UCX/1.19.0/GCCcore-14.3.0/ucx-1.19.0/src/uct/ib/mlx5/rc/rc_mlx5_iface.c:132
 9 0x0000000000050a4a ucs_callbackq_dispatch()  /tmp/vsc40023/easybuild_build/UCX/1.19.0/GCCcore-14.3.0/ucx-1.19.0/src/ucs/datastruct/callbackq.h:215
10 0x0000000000050a4a uct_worker_progress()  /tmp/vsc40023/easybuild_build/UCX/1.19.0/GCCcore-14.3.0/ucx-1.19.0/src/uct/api/uct.h:2823
11 0x0000000000050a4a ucp_worker_progress()  /tmp/vsc40023/easybuild_build/UCX/1.19.0/GCCcore-14.3.0/ucx-1.19.0/src/ucp/core/ucp_worker.c:3059
12 0x00000000002496af mca_pml_ucx_recv()  ???:0
13 0x00000000000d2303 PMPI_Recv()  ???:0
14 0x000000000040d782 gmx::test::(anonymous namespace)::GpuAwareMpiTest_SendFromDeviceBufferToUnpinnedHostBuffer_Test::TestBody()  ???:0
15 0x0000000000062c2d testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>()  ???:0
16 0x000000000004fea6 testing::Test::Run()  ???:0
17 0x0000000000050045 testing::TestInfo::Run()  ???:0
18 0x0000000000050251 testing::TestSuite::Run()  ???:0
19 0x000000000005a24b testing::internal::UnitTestImpl::RunAllTests()  ???:0
20 0x000000000005a845 testing::UnitTest::Run()  ???:0
21 0x000000000040b3fc main()  ???:0
22 0x00000000000295d0 __libc_start_call_main()  ???:0
23 0x0000000000029680 __libc_start_main_alias_2()  :0
24 0x000000000040b535 _start()  ???:0
=================================
[node3904:3487992] *** Process received signal ***
[node3904:3487992] Signal: Aborted (6)
[node3904:3487992] Signal code:  (-6)
[node3904:3487992] [ 0] /lib64/libc.so.6(+0x3ebf0)[0x14cdeb23ebf0]
[node3904:3487992] [ 1] /lib64/libc.so.6(+0x8bedc)[0x14cdeb28bedc]
[node3904:3487992] [ 2] /lib64/libc.so.6(raise+0x16)[0x14cdeb23eb46]
[node3904:3487992] [ 3] /lib64/libc.so.6(abort+0xd3)[0x14cdeb228833]
[node3904:3487992] [ 4] /user/gent/400/vsc40023/eb_scratch/RHEL9/zen3-ampere-ib/software/UCX/1.19.0-GCCcore-14.3.0/lib64/libucs.so.0(+0x2e96b)[0x14cdd883296b]
[node3904:3487992] [ 5] /user/gent/400/vsc40023/eb_scratch/RHEL9/zen3-ampere-ib/software/UCX/1.19.0-GCCcore-14.3.0/lib64/libucs.so.0(+0x3290d)[0x14cdd883690d]
[node3904:3487992] [ 6] /user/gent/400/vsc40023/eb_scratch/RHEL9/zen3-ampere-ib/software/UCX/1.19.0-GCCcore-14.3.0/lib64/libucs.so.0(ucs_log_dispatch+0xdd)[0x14cdd8836c4d]
[node3904:3487992] [ 7] /user/gent/400/vsc40023/eb_scratch/RHEL9/zen3-ampere-ib/software/UCX/1.19.0-GCCcore-14.3.0/lib64/ucx/libuct_ib_mlx5.so.0(uct_ib_mlx5_completion_with_err+0x4de)[0x14cdc81aa9ce]
[node3904:3487992] [ 8] /user/gent/400/vsc40023/eb_scratch/RHEL9/zen3-ampere-ib/software/UCX/1.19.0-GCCcore-14.3.0/lib64/ucx/libuct_ib_mlx5.so.0(uct_rc_mlx5_iface_handle_failure+0x13c)[0x14cdc81bea8c]
[node3904:3487992] [ 9] /user/gent/400/vsc40023/eb_scratch/RHEL9/zen3-ampere-ib/software/UCX/1.19.0-GCCcore-14.3.0/lib64/ucx/libuct_ib_mlx5.so.0(uct_ib_mlx5_check_completion+0x35)[0x14cdc81aba85]
[node3904:3487992] [10] /user/gent/400/vsc40023/eb_scratch/RHEL9/zen3-ampere-ib/software/UCX/1.19.0-GCCcore-14.3.0/lib64/ucx/libuct_ib_mlx5.so.0(+0x2aeb7)[0x14cdc81c0eb7]
[node3904:3487992] [11] /user/gent/400/vsc40023/eb_scratch/RHEL9/zen3-ampere-ib/software/UCX/1.19.0-GCCcore-14.3.0/lib64/libucp.so.0(ucp_worker_progress+0x5a)[0x14cdd88bda4a]
[node3904:3487992] [12] /user/gent/400/vsc40023/eb_scratch/RHEL9/zen3-ampere-ib/software/OpenMPI/5.0.8-GCC-14.3.0/lib64/libmpi.so.40(mca_pml_ucx_recv+0x11f)[0x14cdebb2e6af]
[node3904:3487992] [13] /user/gent/400/vsc40023/eb_scratch/RHEL9/zen3-ampere-ib/software/OpenMPI/5.0.8-GCC-14.3.0/lib64/libmpi.so.40(PMPI_Recv+0x163)[0x14cdeb9b7303]
[node3904:3487992] [14] /tmp/vsc40023/easybuild_build/GROMACS/2026.2/foss-2025b-CUDA-12.9.1/easybuild_obj/bin/gpu_utils-mpi-test[0x40d782]
[node3904:3487992] [15] /user/gent/400/vsc40023/eb_scratch/RHEL9/zen3-ampere-ib/software/GROMACS/2026.2-foss-2025b-CUDA-12.9.1/lib/libgtest.so.1.17.0(_ZN7testing8internal35HandleExceptionsInMethodIfSupportedINS_4TestEvEET0_PT_MS4_FS3_vEPKc+0x3d)[0x14cdebcd7c2d]
[node3904:3487992] [16] /user/gent/400/vsc40023/eb_scratch/RHEL9/zen3-ampere-ib/software/GROMACS/2026.2-foss-2025b-CUDA-12.9.1/lib/libgtest.so.1.17.0(_ZN7testing4Test3RunEv+0xd6)[0x14cdebcc4ea6]
[node3904:3487992] [17] /user/gent/400/vsc40023/eb_scratch/RHEL9/zen3-ampere-ib/software/GROMACS/2026.2-foss-2025b-CUDA-12.9.1/lib/libgtest.so.1.17.0(_ZN7testing8TestInfo3RunEv+0x165)[0x14cdebcc5045]
[node3904:3487992] [18] /user/gent/400/vsc40023/eb_scratch/RHEL9/zen3-ampere-ib/software/GROMACS/2026.2-foss-2025b-CUDA-12.9.1/lib/libgtest.so.1.17.0(_ZN7testing9TestSuite3RunEv+0x201)[0x14cdebcc5251]
[node3904:3487992] [19] /user/gent/400/vsc40023/eb_scratch/RHEL9/zen3-ampere-ib/software/GROMACS/2026.2-foss-2025b-CUDA-12.9.1/lib/libgtest.so.1.17.0(_ZN7testing8internal12UnitTestImpl11RunAllTestsEv+0x65b)[0x14cdebccf24b]
[node3904:3487992] [20] /user/gent/400/vsc40023/eb_scratch/RHEL9/zen3-ampere-ib/software/GROMACS/2026.2-foss-2025b-CUDA-12.9.1/lib/libgtest.so.1.17.0(_ZN7testing8UnitTest3RunEv+0x75)[0x14cdebccf845]
[node3904:3487992] [21] /tmp/vsc40023/easybuild_build/GROMACS/2026.2/foss-2025b-CUDA-12.9.1/easybuild_obj/bin/gpu_utils-mpi-test[0x40b3fc]
[node3904:3487992] [22] /lib64/libc.so.6(+0x295d0)[0x14cdeb2295d0]
[node3904:3487992] [23] /lib64/libc.so.6(__libc_start_main+0x80)[0x14cdeb229680]
[node3904:3487992] [24] /tmp/vsc40023/easybuild_build/GROMACS/2026.2/foss-2025b-CUDA-12.9.1/easybuild_obj/bin/gpu_utils-mpi-test[0x40b535]
[node3904:3487992] *** End of error message ***
--------------------------------------------------------------------------
prterun noticed that process rank 1 with PID 3487992 on node node3904 exited on
signal 6 (Aborted).
--------------------------------------------------------------------------

[boegel]

Setting $UCX_IB_GPU_DIRECT_RDMA to n helps to circumvent the problem, see also https://docs.nvidia.com/nvidia-hpc-x-software-toolkit-rev-2-13-1-lts.pdf:

2235234 Description: On some platforms, GPUDirect RDMA does not work reliably when the path
between HCA and GPU traverses QPI link.
Workaround: Disable GPUDirect support in UCX by setting UCX_IB_GPU_DIRECT_RDMA=n.
Keywords: GPUDirect. RDMA, UCX
Discovered in Version: 2.7 (UCX 1.9)

$ nvidia-smi  --version
NVIDIA-SMI version  : 595.71.05
NVML version        : 595.71
DRIVER version      : 595.71.05
CUDA Version        : 13.2