Skip to content

Commit df29d36

Browse files
gfenoygfenoy
committed
Helm chart and Applicaiton version bump
Update the ZOO-Project Docker image to the latest version Add in-cluster Toil WES enhancements: Celery workers, shared storage options, and optional node-level NFS/EFS hostPath support Update Toil WES documentation and values schema to cover celery/sharedStorage/nfsMount settings and hostPath precedence Wire zookernel.replicaCount into templating and schema so the documented parameter is now effective (fix #27)~
1 parent 4572205 commit df29d36

9 files changed

Lines changed: 281 additions & 21 deletions

File tree

‎zoo-project-dru/Chart.yaml‎

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -23,13 +23,13 @@ type: application
2323
# This is the chart version. This version number should be incremented each time you make changes
2424
# to the chart and its templates, including the app version.
2525
# Versions are expected to follow Semantic Versioning (https://semver.org/)
26-
version: 0.10.3
26+
version: 0.10.4
2727

2828
# This is the version number of the application being deployed. This version number should be
2929
# incremented each time you make changes to the application. Versions are not expected to
3030
# follow Semantic Versioning. They should reflect the version the application is using.
3131
# It is recommended to use it with quotes.
32-
appVersion: "0.2.47"
32+
appVersion: "0.2.48"
3333

3434
keywords:
3535
- eoepca
@@ -80,19 +80,20 @@ dependencies:
8080

8181
annotations:
8282
artifacthub.io/changes: |
83-
- kind: security
84-
description: "Fix security issues in the browse service"
8583
- kind: changed
86-
description: "Update the ZOO-Project Docker image to the latest version to include security patches and improvements"
84+
description: "Update the ZOO-Project Docker image to the latest version"
8785
links:
8886
- name: GitHub commit
89-
url: https://github.com/ZOO-Project/ZOO-Project/commit/8b3d76c
87+
url: https://github.com/ZOO-Project/ZOO-Project/commit/19f3c4e
9088
- kind: added
91-
description: "Add support for Toil WES service with Celery and RabbitMQ integration"
89+
description: "Add in-cluster Toil WES enhancements: Celery workers, shared storage options, and optional node-level NFS/EFS hostPath support"
9290
- kind: changed
93-
description: "Fix versions of curl and calrissian images used in the chart to avoid using latest tags"
91+
description: "Update Toil WES documentation and values schema to cover celery/sharedStorage/nfsMount settings and hostPath precedence"
9492
- kind: changed
95-
description: "Update rabbitmq and postgres images to more recent versions to reduce vulnerabilities"
93+
description: "Wire zookernel.replicaCount into templating and schema so the documented parameter is now effective"
94+
links:
95+
- name: GitHub issue
96+
url: https://github.com/ZOO-Project/charts/issues/27
9697
- kind: changed
9798
description: "Align the README.md with the Chart.yaml version"
9899

‎zoo-project-dru/README.md‎

Lines changed: 26 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ To install the chart with the release name `my-zoo-project-dru`:
2424

2525
````bash
2626
helm repo add zoo-project https://zoo-project.github.io/charts/
27-
helm install my-zoo-project-dru zoo-project/zoo-project-dru --version 0.10.3
27+
helm install my-zoo-project-dru zoo-project/zoo-project-dru --version 0.10.4
2828
````
2929

3030
## Parameters
@@ -719,18 +719,36 @@ In addition to connecting to an external WES endpoint (for example HPC), this ch
719719
| toilWes.enabled | Enable in-cluster Toil WES deployment | false |
720720
| toilWes.replicaCount | Number of Toil WES pods | 1 |
721721
| toilWes.serviceAccountName | ServiceAccount name (defaults to `<release>-processing-manager`) | "" |
722+
| toilWes.celery.enabled | Enable Celery mode for Toil WES (`--bypass_celery` is disabled when true) | false |
723+
| toilWes.celery.replicaCount | Number of Celery worker pods | 1 |
724+
| toilWes.celery.brokerUrl | Optional explicit Celery broker URL (`TOIL_WES_BROKER_URL`) | "" |
725+
| toilWes.celery.resultBackend | Celery result backend (`TOIL_WES_RESULT_BACKEND`) | "rpc://" |
726+
| toilWes.celery.workerLogLevel | Celery worker log level | "INFO" |
722727
| toilWes.awsRegion | AWS region used by the Toil AWS-style job store locator | "us-east-1" |
723728
| toilWes.batchSystem | Toil batch system backend | "kubernetes" |
724729
| toilWes.workDir | Toil WES work directory | "/workflows" |
725730
| toilWes.stateStore | Toil WES state store path | "/workflows/state_store" |
731+
| toilWes.sharedStorage.enabled | Mount shared RWX storage in Toil WES (and Celery workers when enabled) | false |
732+
| toilWes.sharedStorage.create | Create the shared PVC automatically | true |
733+
| toilWes.sharedStorage.existingClaim | Existing PVC name (used when `create=false`) | "" |
734+
| toilWes.sharedStorage.mountPath | Mount path of shared storage in containers | "/workflows" |
735+
| toilWes.sharedStorage.accessMode | Access mode for shared PVC | "ReadWriteMany" |
736+
| toilWes.sharedStorage.size | Shared PVC requested size | "20Gi" |
737+
| toilWes.sharedStorage.storageClass | Shared PVC storage class (`""` lets the cluster default apply) | "" |
726738
| toilWes.privileged | Enable privileged mode for Kubernetes worker pods launched by Toil (`--kubernetesPrivileged`) | true |
727739
| toilWes.serviceAccount | Optional dedicated ServiceAccount value kept for compatibility (currently not wired into `--kubernetesServiceAccount`, which uses `toilWes.serviceAccountName`) | "" |
728-
| toilWes.hostPath | Optional hostPath mounted by Toil workers (`TOIL_KUBERNETES_HOST_PATH`) | "" |
740+
| toilWes.hostPath | Explicit hostPath for Toil workers (`TOIL_KUBERNETES_HOST_PATH`) | "" |
741+
| toilWes.nfsMount.enabled | Enable node-level NFS/EFS mounting DaemonSet for Toil hostPath usage | false |
742+
| toilWes.nfsMount.server | NFS/EFS server hostname or IP | "" |
743+
| toilWes.nfsMount.path | Exported NFS path on server | "/" |
744+
| toilWes.nfsMount.nodeMountPath | Path mounted on each Kubernetes node (used as fallback hostPath) | "/mnt/toil-shared" |
745+
| toilWes.nfsMount.mountOptions | NFS mount options used by node-level mount and static PV | ["nfsvers=4.1","rsize=1048576","wsize=1048576","hard","timeo=600","retrans=2"] |
746+
| toilWes.nfsMount.image.repository | Image repository used by NFS/EFS node mounter DaemonSet | "alpine" |
747+
| toilWes.nfsMount.image.tag | Image tag used by NFS/EFS node mounter DaemonSet | "3.21" |
748+
| toilWes.nfsMount.image.pullPolicy | Image pull policy for NFS/EFS node mounter DaemonSet | "IfNotPresent" |
729749
| toilWes.enable_cache | Enable shared Singularity/Apptainer cache variables for Toil workers | false |
730750
| toilWes.image.repository | Toil image repository | "quay.io/ucsc_cgl/toil" |
731751
| toilWes.image.tag | Toil image tag | "9.4.1-c6a39f2c827899f9a1dd0018ca2b72c675120096-py3.13" |
732-
| toilWes.command | Container command | ["toil"] |
733-
| toilWes.args | Container args | ["server", "--host=0.0.0.0", "--port=8080"] |
734752
| toilWes.service.type | Service type | "ClusterIP" |
735753
| toilWes.service.port | Toil WES service port | 8080 |
736754
| toilWes.s3.realaws | Use real AWS S3 instead of a local S3-compatible endpoint | true |
@@ -750,6 +768,10 @@ In addition to connecting to an external WES endpoint (for example HPC), this ch
750768
| toilWes.tolerations | Pod tolerations for the Toil WES deployment | [] |
751769
| toilWes.affinity | Pod affinity rules for the Toil WES deployment | {} |
752770

771+
> **Note**: `TOIL_KUBERNETES_HOST_PATH` resolution order is: `toilWes.hostPath` (if set) > `toilWes.nfsMount.nodeMountPath` (if `nfsMount.enabled=true`) > unset.
772+
773+
> **Note**: when `toilWes.sharedStorage.storageClass` is set, the chart does not create the static NFS PV (`pv-toil-wes-nfs.yaml`) and relies on dynamic provisioning via your StorageClass.
774+
753775
> **Note**: when deploying on a local minikube cluster, you may need to enable the metrics-server addon with the following command: `minikube addons enable metrics-server`.
754776

755777

‎zoo-project-dru/templates/claim-toil-wes-shared.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,5 +13,11 @@ spec:
1313
storage: {{ .Values.toilWes.sharedStorage.size }}
1414
{{- if ne .Values.toilWes.sharedStorage.storageClass "" }}
1515
storageClassName: {{ .Values.toilWes.sharedStorage.storageClass | quote }}
16+
{{- else if .Values.toilWes.nfsMount.enabled }}
17+
# Static binding to the NFS PV created by pv-toil-wes-nfs.yaml
18+
storageClassName: ""
19+
selector:
20+
matchLabels:
21+
zoo-project-dru/pv-binding: {{ .Release.Name }}-toil-wes-nfs
1622
{{- end }}
1723
{{- end }}

‎zoo-project-dru/templates/dp-toil-wes.yaml‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -85,8 +85,13 @@ spec:
8585
--opt=--setEnv=TOIL_AWS_SECRET_NAME={{ .Values.toilWes.s3.credentialsSecret.name }} \
8686
--opt=--setEnv=TOIL_KUBERNETES_OWNER=zoo \
8787
--opt=--setEnv=TOIL_KUBERNETES_PRIVILEGED={{ ternary "True" "False" .Values.toilWes.privileged }} \
88-
{{- if .Values.toilWes.hostPath }}
89-
--opt=--setEnv=TOIL_KUBERNETES_HOST_PATH={{ .Values.toilWes.hostPath }} \
88+
{{- /* explicit hostPath > nodeMountPath NFS > nothing */}}
89+
{{- $hostPath := .Values.toilWes.hostPath }}
90+
{{- if and (not $hostPath) .Values.toilWes.nfsMount.enabled }}
91+
{{- $hostPath = .Values.toilWes.nfsMount.nodeMountPath }}
92+
{{- end }}
93+
{{- if $hostPath }}
94+
--opt=--setEnv=TOIL_KUBERNETES_HOST_PATH={{ $hostPath }} \
9095
{{- end }}
9196
{{- if .Values.toilWes.enable_cache }}
9297
--opt=--setEnv=CWL_SINGULARITY_CACHE={{ .Values.toilWes.workDir | default "/workflows" }}/singularity-cache \

‎zoo-project-dru/templates/dp-zookernel.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ metadata:
55
labels:
66
{{- include "zoo-project-dru.labels" . | nindent 4 }}
77
spec:
8-
replicas: 1
8+
replicas: {{ .Values.zookernel.replicaCount }}
99
selector:
1010
matchLabels:
1111
{{- include "zoo-project-dru.selectorLabels" . | nindent 6 }}

‎zoo-project-dru/templates/ds-toil-wes-nfs-mounter.yaml‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
{{- if and .Values.toilWes.enabled .Values.toilWes.nfsMount.enabled -}}
2+
{{- if not .Values.toilWes.nfsMount.server -}}
3+
{{- fail "toilWes.nfsMount.server is required when toilWes.nfsMount.enabled is true" -}}
4+
{{- end -}}
5+
# DaemonSet that mounts the NFS/EFS share on every Kubernetes node.
6+
# This allows Toil worker pods to access the shared directory via hostPath
7+
# (TOIL_KUBERNETES_HOST_PATH = toilWes.nfsMount.nodeMountPath).
8+
#
9+
# Prerequisite: nodes must have network access to the NFS/EFS server.
10+
# For AWS EFS, use the file system DNS endpoint (fs-xxxx.efs.<region>.amazonaws.com).
11+
apiVersion: apps/v1
12+
kind: DaemonSet
13+
metadata:
14+
name: {{ .Release.Name }}-toil-nfs-mounter
15+
labels:
16+
{{- include "zoo-project-dru.labels" . | nindent 4 }}
17+
app.kubernetes.io/component: toil-nfs-mounter
18+
spec:
19+
selector:
20+
matchLabels:
21+
app.kubernetes.io/name: {{ include "zoo-project-dru.name" . }}-toil-nfs-mounter
22+
app.kubernetes.io/instance: {{ .Release.Name }}-toil-nfs-mounter
23+
updateStrategy:
24+
type: RollingUpdate
25+
template:
26+
metadata:
27+
labels:
28+
app.kubernetes.io/name: {{ include "zoo-project-dru.name" . }}-toil-nfs-mounter
29+
app.kubernetes.io/instance: {{ .Release.Name }}-toil-nfs-mounter
30+
spec:
31+
# hostPID allows nsenter to operate in the mount namespace
32+
# of the node init process (PID 1), which mounts NFS directly on the node.
33+
hostPID: true
34+
# Tolerate all taints to ensure the DaemonSet runs on all nodes
35+
tolerations:
36+
- operator: Exists
37+
{{- with .Values.toilWes.nodeSelector }}
38+
nodeSelector:
39+
{{- toYaml . | nindent 8 }}
40+
{{- end }}
41+
initContainers:
42+
- name: nfs-mount
43+
image: "{{ .Values.toilWes.nfsMount.image.repository }}:{{ .Values.toilWes.nfsMount.image.tag }}"
44+
imagePullPolicy: {{ .Values.toilWes.nfsMount.image.pullPolicy }}
45+
securityContext:
46+
privileged: true
47+
command:
48+
- sh
49+
- -c
50+
- |
51+
set -e
52+
NFS_SERVER="{{ .Values.toilWes.nfsMount.server }}"
53+
NFS_PATH="{{ .Values.toilWes.nfsMount.path }}"
54+
MOUNT_POINT="{{ .Values.toilWes.nfsMount.nodeMountPath }}"
55+
MOUNT_OPTIONS="{{ .Values.toilWes.nfsMount.mountOptions | join "," }}"
56+
57+
# Install nfs-utils in the Alpine container to provide mount.nfs
58+
apk add --no-cache nfs-utils > /dev/null 2>&1
59+
60+
# Create the mount point on the host node via nsenter
61+
nsenter --mount=/proc/1/ns/mnt -- mkdir -p "${MOUNT_POINT}"
62+
63+
# Mount the NFS share on the host node if it is not already mounted
64+
if nsenter --mount=/proc/1/ns/mnt -- mountpoint -q "${MOUNT_POINT}"; then
65+
echo "INFO: ${MOUNT_POINT} is already mounted on the node."
66+
else
67+
echo "INFO: Mounting ${NFS_SERVER}:${NFS_PATH} on ${MOUNT_POINT} (options: ${MOUNT_OPTIONS})"
68+
nsenter --mount=/proc/1/ns/mnt -- mount \
69+
-t nfs \
70+
-o "${MOUNT_OPTIONS}" \
71+
"${NFS_SERVER}:${NFS_PATH}" \
72+
"${MOUNT_POINT}"
73+
echo "INFO: Mount successful."
74+
fi
75+
containers:
76+
- name: keepalive
77+
image: "{{ .Values.toilWes.nfsMount.image.repository }}:{{ .Values.toilWes.nfsMount.image.tag }}"
78+
imagePullPolicy: {{ .Values.toilWes.nfsMount.image.pullPolicy }}
79+
securityContext:
80+
privileged: true
81+
command:
82+
- sh
83+
- -c
84+
- |
85+
NFS_SERVER="{{ .Values.toilWes.nfsMount.server }}"
86+
NFS_PATH="{{ .Values.toilWes.nfsMount.path }}"
87+
MOUNT_POINT="{{ .Values.toilWes.nfsMount.nodeMountPath }}"
88+
MOUNT_OPTIONS="{{ .Values.toilWes.nfsMount.mountOptions | join "," }}"
89+
90+
# Cleanly unmount when the pod is stopping
91+
trap 'echo "INFO: Unmounting ${MOUNT_POINT}..."; \
92+
nsenter --mount=/proc/1/ns/mnt -- umount "${MOUNT_POINT}" 2>/dev/null || true; \
93+
exit 0' TERM INT
94+
95+
echo "INFO: NFS active: ${NFS_SERVER}:${NFS_PATH} -> ${MOUNT_POINT}"
96+
97+
# Monitoring loop: remount if the mount disappears
98+
while true; do
99+
if ! nsenter --mount=/proc/1/ns/mnt -- mountpoint -q "${MOUNT_POINT}"; then
100+
echo "WARN: Mount lost, attempting remount..."
101+
nsenter --mount=/proc/1/ns/mnt -- mount \
102+
-t nfs \
103+
-o "${MOUNT_OPTIONS}" \
104+
"${NFS_SERVER}:${NFS_PATH}" \
105+
"${MOUNT_POINT}" || echo "ERROR: Remount failed."
106+
fi
107+
sleep 60 &
108+
wait $!
109+
done
110+
resources:
111+
requests:
112+
cpu: 10m
113+
memory: 16Mi
114+
limits:
115+
cpu: 100m
116+
memory: 64Mi
117+
{{- end }}

‎zoo-project-dru/templates/pv-toil-wes-nfs.yaml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
{{- if and .Values.toilWes.enabled .Values.toilWes.nfsMount.enabled .Values.toilWes.sharedStorage.enabled .Values.toilWes.sharedStorage.create (eq (.Values.toilWes.sharedStorage.storageClass | default "") "") -}}
2+
# Static NFS PersistentVolume for the toil-wes pod.
3+
# This PV is created only when both nfsMount and sharedStorage are enabled,
4+
# and allows the toil-wes pod itself to mount the same NFS share as the nodes.
5+
# The PVC defined in claim-toil-wes-shared.yaml will bind to this PV via the label selector.
6+
apiVersion: v1
7+
kind: PersistentVolume
8+
metadata:
9+
name: {{ .Release.Name }}-toil-wes-nfs
10+
labels:
11+
{{- include "zoo-project-dru.labels" . | nindent 4 }}
12+
app.kubernetes.io/component: toil-nfs-pv
13+
# Label used by the PVC for static binding
14+
zoo-project-dru/pv-binding: {{ .Release.Name }}-toil-wes-nfs
15+
spec:
16+
capacity:
17+
storage: {{ .Values.toilWes.sharedStorage.size }}
18+
accessModes:
19+
- {{ .Values.toilWes.sharedStorage.accessMode }}
20+
persistentVolumeReclaimPolicy: Retain
21+
{{- if ne .Values.toilWes.sharedStorage.storageClass "" }}
22+
storageClassName: {{ .Values.toilWes.sharedStorage.storageClass | quote }}
23+
{{- else }}
24+
storageClassName: ""
25+
{{- end }}
26+
mountOptions:
27+
{{- toYaml .Values.toilWes.nfsMount.mountOptions | nindent 4 }}
28+
nfs:
29+
server: {{ .Values.toilWes.nfsMount.server | quote }}
30+
path: {{ .Values.toilWes.nfsMount.path | quote }}
31+
readOnly: false
32+
{{- end }}

0 commit comments

Comments
 (0)