bench: rewrite latency perf test into a ping-pong (backend-generic support)

[dssgabriel]

Description

This PR is a first draft of a backend-generic ping-pong latency benchmark for Kokkos Comm, fixing #244.

It replaces bench.mpi.osu_latency with a new bench.core.latency perf test that compares the Kokkos Comm APIs against direct backend calls (both MPI and NCCL).

One important difference from the existing MPI-only implementation in perf_tests/mpi/test_osu_latency.cpp is that this version measures a true ping-pong: rank 0 sends to rank 1, then receives the response. The existing MPI benchmark measured only one-way latency.

• Related issue(s): Fix OSU latency benchmarks to actually measure latency #244
• Depends on: none

Changes

• Affected areas: perf tests, build
• Breaking change(s)? no

List:

• Add backend-generic latency benchmark in perf_tests/bench_latency.cpp.
• Rewrite shared do_iteration helper in perf_tests/utils.hpp with high_resolution_clock instead of steady_clock, and remove MPI_Comm param.
• Wire the new benchmark into perf_tests/CMakeLists.txt as bench.core.latency.
• Update kc_add_perf_test so core perf tests can link the same Kokkos Comm/MPI/NCCL support used by core unit tests (ugly but avoids duplicating NCCL init logic across unit and perf tests).

Checklist

• Tests are up-to-date
• Documentation is up-to-date

[dssgabriel]

@nicoleavans This is the PR with the backend-agnostic OSU lat implementation. For some reason, I cannot add you as a reviewer 🤷‍♂️

And here are the results we've obtained on a GH200 + Bull eXascale Interconnect v2 (BXI) system:

[cedricchevalier19]

Looks ok, just a few comments.

[cedricchevalier19]

You can add a check or change must be by is.

[cedricchevalier19]

Can we group a little bit the post treatment?
->Unit(benchmark::kMicrosecond)->RangeMultiplier(8)->Range(1, 1 << 28); is quite heavy to read.

[dssgabriel]

I agree, will look into how we can achieve this.

Alternatively, we may consider porting all our test/benchmark infrastructure to a more modern framework, such as Catch2. We definitely need to rethink how we validate multi-rank tests anyway.

[cedricchevalier19]

I am not a huge fan of using FetchContent. Can we test if it is available locally or can we avoid making it mandatory ?

[cedricchevalier19]

Ideally can we keep all ? Or can we at least compute the average too ?

[dssgabriel]

Not sure if we can keep all (I don't know if it's possible to attach an array of results to the custom counters), but we can certainly compute the average (as well as min, median, stddev, etc.).

Note that the average isn't necessarily meaningful here, since it is the average latency of all ranks. I had tried to report all these metrics at some point, but the results varied so much across ranks that I didn't see the point in keeping them, so I reverted everything back to how it is here.

[dssgabriel]

Thanks for the review @cedricchevalier19.

I think this still needs quite a bit of work before we can merge it into Kokkos Comm, and I am not sure the "OSU latency" naming fits.
Specifically, OSU latency measures ping-pong, so all calls should have blocking semantics: rank 1 needs to actually receive rank 0's message before responding. The time for the round trip divided by two is the latency. That is not what the benchmarks in this PR implement, nor what the existing KokkosComm::mpi-based benchmarks do.
We should have a proper latency benchmark implementation and also consider adding a suite of bandwidth benchmarks (for P2P, AG, AR and A2A).

[dssgabriel]

Commands to run config/build/run this benchmark:

# Load dependencies via `module` or `spack`
spack load cmake cuda mpi nccl

# Fetch + configure + build + install Kokkos (if not available on your machine, else load via module/spack directly)
export KOKKOS_VERSION=5.1.1
export KOKKOS_DOWNLOAD_URL=https://github.com/kokkos/kokkos/releases/download/${KOKKOS_VERSION}
mkdir -p _deps && cd _deps && curl -sLO ${KOKKOS_DOWNLOAD_URL}/kokkos-${KOKKOS_VERSION}.tar.gz && cd ..
tar xzf _deps/kokkos-${KOKKOS_VERSION}.tar.gz -C _deps/
cmake -S _deps/kokkos-${KOKKOS_VERSION} -B _deps/kokkos-${KOKKOS_VERSION}-build -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS="-mcpu=native" -DKokkos_ENABLE_CUDA=ON -DKokkos_ARCH_<SYSTEM_ARCH>=ON
cmake --build _deps/kokkos-${KOKKOS_VERSION}-build -j
cmake --install _deps/kokkos-${KOKKOS_VERSION}-build --prefix _deps/kokkos-${KOKKOS_VERSION}-install

# Configure Kokkos Comm with just-installed Kokkos
cmake -B build -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS="-mcpu=native" -DCMAKE_CXX_COMPILER="$(pwd)/_deps/kokkos-${KOKKOS_VERSION}/bin/nvcc_wrapper" -DKokkos_ROOT="$(pwd)/_deps/kokkos-${KOKKOS_VERSION}-install" -DKokkosComm_ENABLE_NCCL=ON -DKokkosComm_ENABLE_PERFTESTS=ON

# Build
cmake --build build --target bench.core.latency -j

# Run the benchmark (via ctest)
ctest --test-dir build -R bench.core.latency
# (or manually)
mpirun -n 2 build/perf_tests/bench.core.latency
# (or via a job scheduler for internode/intranode; here assuming GH200 core counts to see the GPUs correctly)
srun -p <PARTITION> -N2 -n2 -c72 build/perf_tests/bench.core.latency
srun -p <PARTITION> -N1 -n2 -c144 build/perf_tests/bench.core.latency

[cedricchevalier19]

Perhaps we should add another comm to measure a ping from 1 :

sequenceDiagram
    0->>1: send1
    activate 0
    1->>0: send2
    deactivate 0
    activate 1
     0->>1: send 3
     deactivate 1

Loading

We time the grey boxes.

[dssgabriel]

Good idea. How do we report that, though? Currently, only rank 0 writes the results (hence we reported "iteration" time as the max on all ranks).