Skip to content

Commit 986513b

Browse files
author
Juan Carlos Graciosa
committed
Add Gadi Singularity container build files for UW3
1 parent 0dcd61f commit 986513b

3 files changed

Lines changed: 480 additions & 0 deletions

File tree

‎docs/developer/gadi_singularity/README.md‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# Building Underworld3 for Gadi (NCI)
2+
3+
This directory contains two Containerfiles to build the Underworld3 (UW3) Singularity image for Gadi (nci.org.au).
4+
5+
Both use Rocky Linux 8.10 to match Gadi's OS for ABI compatibility.
6+
7+
## Build Order
8+
9+
Build commands must be run from the top-level `underworld3/` directory (the build context).
10+
Builds targeting Gadi must use `--platform linux/amd64`.
11+
12+
### 1. Build PETSc layer
13+
14+
```bash
15+
podman build . \
16+
--platform linux/amd64 \
17+
--format docker \
18+
-t ghcr.io/<user>/petsc:3.25.0-ompi \
19+
-f ./docs/developer/gadi_singularity/petsc.rhel
20+
```
21+
22+
### 2. Push PETSc image to registry
23+
24+
```bash
25+
podman push ghcr.io/<user>/petsc:3.25.0-ompi
26+
```
27+
28+
### 3. Build Underworld3
29+
30+
```bash
31+
podman build . \
32+
--platform linux/amd64 \
33+
--format docker \
34+
--build-arg PETSC_IMAGE=ghcr.io/<user>/petsc:3.25.0-ompi \
35+
--build-arg UW3_BRANCH=development \
36+
-t ghcr.io/<user>/underworld3-gadi:latest \
37+
-f ./docs/developer/gadi_singularity/underworld3.rhel
38+
```
39+
40+
### 4. Push Underworld3 image
41+
42+
```bash
43+
podman push ghcr.io/<user>/underworld3-gadi:latest
44+
```
45+
46+
## What Each File Does
47+
48+
- **petsc.rhel** — Builds PETSc 3.25.0 with full AMR support (petsc4py, slepc4py, mmg, parmmg, etc.)
49+
- **underworld3.rhel** — Builds Underworld3 on top of the PETSc image
50+
51+
## Running on Gadi
52+
53+
Pull the image on Gadi (redirect cache to scratch to avoid home quota issues):
54+
55+
```bash
56+
export SINGULARITY_CACHEDIR=/scratch/<project>/<user>/.singularity
57+
module load singularity
58+
singularity pull docker://ghcr.io/<user>/underworld3-gadi:latest
59+
```
60+
61+
Run a script with MPI:
62+
63+
```bash
64+
module load singularity
65+
module load openmpi/4.1.7
66+
mpiexec -n <ncpus> singularity exec underworld3-gadi_latest.sif python3 <script.py>
67+
```
68+
69+
## Notes
70+
71+
- OpenFabrics (mlx5_0) warnings in the job error log are harmless
72+
- PostHog telemetry failures on compute nodes are harmless (no outbound internet)
73+
- The ghcr.io images must be set to **public** for Singularity to pull without authentication

‎docs/developer/gadi_singularity/petsc.rhel‎

Lines changed: 197 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,197 @@
1+
#####################################################################
2+
# UW3 PETSc container
3+
# Multi stage Containerfile based on UW2 version
4+
# This builds PETSc according to the pixi amr-dev environment
5+
# see https://docs.docker.com/get-started/docker-concepts/building-images/multi-stage-builds/
6+
#
7+
# Stages:
8+
# 1. 'runtime'
9+
# The runtime environment (packages, permissions, ENV vars.)
10+
# is consistent accross all stages of this Containerfile.
11+
#
12+
# 2. 'builder'
13+
# The builder layer, takes the runtime layer and add compiling / building software
14+
# that is added to /usr/local and /opt/venv
15+
#
16+
# 3. 'final' == runtime + min. builder
17+
# The final image is a composite of the runtime layer and the
18+
# minimal sections of the builder layer's final software stack.
19+
#
20+
# To build use podman from the top level underworld directory. i.e.
21+
# $ podman build . \
22+
# --platform linux/amd64 \
23+
# --format docker \
24+
# -t new_image_name \
25+
# -f ./docs/development/gadi_singularity/petsc.rhel
26+
#####################################################################
27+
28+
# The following are passed in via --build-args
29+
# Must go before the 1st FROM see
30+
# https://docs.docker.com/engine/reference/builder/#understand-how-arg-and-from-interact
31+
ARG PYTHON_VERSION="3.12"
32+
ARG PETSC_VERSION="3.25.0"
33+
ARG BASE_IMAGE="quay.io/rockylinux/rockylinux:8.10"
34+
35+
# 1. Stage 1: 'runtime'
36+
FROM ${BASE_IMAGE} as runtime
37+
LABEL maintainer="https://github.com/underworldcode/"
38+
39+
# need to repeat ARGS after every FROM
40+
ARG PYTHON_VERSION
41+
42+
#### Containerfile ENV vars - for all image stages
43+
ENV LANG=C.UTF-8
44+
ENV PYVER=${PYTHON_VERSION}
45+
46+
# gadi-specific settings
47+
ENV OPENBLAS_NUM_THREADS=1
48+
ENV OMPI_MCA_io=ompio
49+
50+
# add user jovyan
51+
ENV NB_USER jovyan
52+
ENV NB_HOME /home/$NB_USER
53+
RUN useradd -m -s /bin/bash -N $NB_USER
54+
55+
RUN yum update -y \
56+
&& yum install -y \
57+
bash-completion \
58+
openssh \
59+
openblas \
60+
python${PYVER}-pip \
61+
python${PYVER}-devel \
62+
openmpi \
63+
findutils \
64+
&& yum clean all \
65+
&& rm -rf /var/cache/yum
66+
67+
# add system openmpi to $PATH and $LD_LIBRARY_PATH
68+
ENV PATH=/usr/lib64/openmpi/bin:$PATH
69+
ENV LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
70+
71+
ENV PYOPT=/opt/venv
72+
# build and set open permissions on virtual environment
73+
RUN python${PYVER} -m venv $PYOPT \
74+
&& chmod ugo+rwx $PYOPT
75+
76+
# define python env vars.
77+
# prepappending on PATH means all pip install will goto the PYOPT
78+
ENV PATH=$PYOPT/bin:$PATH
79+
ENV PYTHONPATH=$PYTHONPATH:$PYOPT/lib/python${PYVER}/site-packages
80+
81+
# runtime python requirements
82+
RUN python${PYVER} -m pip install wheel \
83+
"numpy<2"
84+
85+
# 2. Define the builder layer
86+
FROM runtime as builder
87+
88+
ARG PETSC_VERSION
89+
ARG PYTHON_VERSION
90+
91+
RUN yum install -y \
92+
ca-certificates \
93+
wget \
94+
make \
95+
gcc \
96+
gcc-gfortran \
97+
gcc-c++ \
98+
cmake \
99+
patch \
100+
openblas \
101+
zlib-devel \
102+
openmpi-devel \
103+
findutils \
104+
git \
105+
flex \
106+
bison \
107+
&& yum clean all \
108+
&& rm -rf /var/cache/yum
109+
# NOTE flex and bison are needed to build
110+
111+
RUN python${PYVER} -m pip install "cython>=3.1" \
112+
"setuptools>=75" \
113+
"meson" \
114+
"meson-python" \
115+
"ninja" \
116+
&& python${PYVER} -m pip install --no-cache-dir --no-binary=mpi4py "mpi4py>=4,<5"
117+
# no-binary for mpi4py to force build against openmpi, rather than mpich (default)
118+
119+
# copy patches into builder
120+
COPY petsc-custom/patches/scotch-7.0.10-c23-fix.tar.gz /tmp/scotch.tar.gz
121+
COPY petsc-custom/patches/plexfem-internal-boundary-ownership-fix.patch /tmp/
122+
123+
# get petsc
124+
RUN mkdir -p /tmp/src
125+
WORKDIR /tmp/src
126+
RUN wget https://web.cels.anl.gov/projects/petsc/download/release-snapshots/petsc-lite-${PETSC_VERSION}.tar.gz --no-check-certificate \
127+
&& tar -zxf petsc-lite-${PETSC_VERSION}.tar.gz
128+
WORKDIR /tmp/src/petsc-${PETSC_VERSION}
129+
130+
# apply patch then configure
131+
# patch may already be included in newer PETSc versions - skip gracefully if it doesn't apply
132+
RUN if patch -p1 --dry-run < /tmp/plexfem-internal-boundary-ownership-fix.patch 2>/dev/null; then \
133+
patch -p1 < /tmp/plexfem-internal-boundary-ownership-fix.patch; \
134+
echo "plexfem patch applied successfully"; \
135+
else \
136+
echo "plexfem patch not applicable (may already be in this PETSc version), skipping"; \
137+
fi
138+
RUN python${PYVER} ./configure \
139+
--with-debugging=0 \
140+
--prefix=/usr/local \
141+
--with-shared-libraries=1 \
142+
--with-cxx-dialect=C++11 \
143+
"--COPTFLAGS=-g -O3" "--CXXOPTFLAGS=-g -O3" "--FOPTFLAGS=-g -O3" \
144+
--useThreads=0 \
145+
--with-x=0 \
146+
--with-pragmatic=1 \
147+
--with-petsc4py=1 \
148+
--with-slepc4py=1 \
149+
--download-eigen=1 \
150+
--download-metis=1 \
151+
--download-parmetis=1 \
152+
--download-mumps=1 \
153+
--download-scalapack=1 \
154+
--download-hypre=1 \
155+
--download-superlu=1 \
156+
--download-superlu_dist=1 \
157+
--download-mmg=1 \
158+
"--download-mmg-cmake-arguments=-DMMG_INSTALL_PRIVATE_HEADERS=ON -DUSE_SCOTCH=OFF" \
159+
--download-parmmg=1 \
160+
--download-pragmatic=1 \
161+
"--download-ptscotch=/tmp/scotch.tar.gz" \
162+
--download-slepc=1 \
163+
--download-hdf5=1 \
164+
--download-fblaslapack=1 \
165+
--download-zlib=1 \
166+
--download-ctetgen=1 \
167+
--download-triangle=1 \
168+
--with-make-np=2
169+
RUN make PETSC_DIR=`pwd` PETSC_ARCH=arch-linux-c-opt all
170+
RUN make PETSC_DIR=`pwd` PETSC_ARCH=arch-linux-c-opt install \
171+
|| (echo "=== petsc4py build log ===" && \
172+
cat arch-linux-c-opt/lib/petsc/conf/petsc4py.build.log 2>/dev/null && \
173+
echo "=== slepc4py build log ===" && \
174+
cat arch-linux-c-opt/lib/petsc/conf/slepc4py.build.log 2>/dev/null && \
175+
exit 1)
176+
RUN rm -rf /usr/local/share/petsc
177+
178+
# record builder stage packages used
179+
RUN python${PYVER} -m pip freeze > /opt/requirements.txt \
180+
&& dnf history userinstalled > /opt/packages.txt
181+
182+
# Stage 3: 'final'
183+
FROM runtime as final
184+
185+
COPY --from=builder /opt /opt
186+
COPY --from=builder /usr/local /usr/local
187+
188+
# MUST set PETSc environment variables
189+
ENV PETSC_DIR=/usr/local
190+
ENV PYTHONPATH=$PYTHONPATH:$PETSC_DIR/lib
191+
192+
# switch to not-root user and workspace
193+
USER $NB_USER
194+
WORKDIR $NB_HOME
195+
196+
# default command is to run jupyter lab
197+
CMD ["jupyter-lab", "--no-browser", "--ip='0.0.0.0'"]

0 commit comments

Comments
 (0)