Update OSKAR>2.11.1 for Grace Hopper support.

[d3v-null]

Oskar 2.8.3 does not support cuda arch 9.0 required for optimal performance on Grace Hopper GPUs, but it was fixed in 2.11.1

https://github.com/OxfordSKA/OSKAR/blob/e07c9c81c375ec45e4118f49b397009c737606ec/cmake/oskar_compiler_options.cmake#L248

I think Hoppers will still run 86, just sub-optimally

[d3v-null]

I was not able to run OSKAR compiled with cuda compute 86 on Hopper GPUs.

I built a multi-arch Karabo image that runs on arm64 and amd64 ( Dockerfile based on cuda 12.2.2 and ubuntu 22.04 ), and ran various tests in a shell on two different CSCS clusters:

• Besso A100 (amd64)

ssh besso
srun --pty --partition=a100 --gpus-per-task=1 --ntasks-per-node=1 --nodes=1 podman run --rm -it --gpus=all docker://ghcr.io/d3v-null/sp5505:sha-ec41d27-pass bash -l

• Daint.Alps GH200 (arm64)

ssh daint
srun --pty --partition=normal --account=sk014 --gpus-per-task=1 --ntasks-per-node=1 --nodes=1 podman run --rm -it --gpus=all docker://ghcr.io/d3v-null/sp5505:sha-ec41d27-pass bash -l

in both shells, I ran

cd /tmp
curl -Lo GGSM_updated.fits 'https://github.com/GLEAM-X/GLEAM-X-pipeline/raw/master/models/GGSM_updated.fits'
/opt/Karabo-Pipeline/karabo/examples/generate_visibility.py --use-gpus --telescope=MWA --telescope-version=1 --mid-frequency=170e6 --num-channels=16 --time-resolution=8 --frequency-resolution=80e3 --num-timesteps=10 --num-channels=16 --sky-model=/tmp/GGSM_updated.fits --model-cutoff=12.5 --model-count=2000 --model-flux-limit=0.001 --phase-center-ra=150.12 --phase-center-dec=2.21 --out=/tmp/mwa-ph1-10x8s-16x80khz.ms
taql 'select MESSAGE from /tmp/mwa-ph1-10x8s-16x80khz.ms/HISTORY'

On besso, the simulation completes, and a message is printed showing that NVIDIA A100-SXM4-80GB GPUs were used.

On daint, the simulation segfaults:

E|                                                                   
E|== ERROR: Kernel 'mem_set_value_real_r_float' launch failure (CUDA error 209: no kernel image is available for execution on the device).
E|                                                                   
[497b4540f376:00097] *** Process received signal ***
[497b4540f376:00097] Signal: Segmentation fault (11)
[497b4540f376:00097] Signal code: Address not mapped (1)
[497b4540f376:00097] Failing at address: (nil)
[497b4540f376:00097] [ 0] linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0x4000118b07dc]
[497b4540f376:00097] [ 1] /opt/view/lib/liboskar.so.2.8.3(oskar_station_create_copy+0x410)[0x40001924c2d0]
[497b4540f376:00097] [ 2] /opt/view/lib/liboskar.so.2.8.3(oskar_telescope_create_copy+0x1cc)[0x40001923829c]
[497b4540f376:00097] [ 3] /opt/view/lib/liboskar.so.2.8.3(+0x69e98)[0x4000191d9e98]
[497b4540f376:00097] [ 4] E| /lib/aarch64-linux-gnu/libc.so.6(+0x80398)[0x400011d40398]
 [497b4540f376:00097]  [ 5]       E|  /lib/aarch64-linux-gnu/libc.so.6(+0xe9e9c)[0x400011da9e9c]
 [497b4540f376:00097] *** End of error message ***
                                                       Segmentation fault (core dumped)

similarly, when running the Karabo test suite, any test that makes use of OSKAR for simulation has a segfault on Daint with GPUs enabled, but passes (except for a numerical error) with GPUs disabled. The tests pass on Besso with GPUs

pytest -v /opt/Karabo-Pipeline

Although several tests that don't involve OSKAR, but do make use of gpu-enabled software like wsclean (which I was able to compile for cuda compute 90) passed, which is promising.

[anawas]

If you want to reproduce this on daint.alps don't forget to create a valid Container Storage configuration file in your home directory, according to the following minimal template:

$HOME/.config/containers/storage.conf

[storage]
driver = "overlay"
runroot = "/dev/shm/$USER/runroot"
graphroot = "/dev/shm/$USER/root"

See https://docs.cscs.ch/build-install/containers/