fix(manifests): add NVML init container for NVIDIA GPU Operator support

Closes #2484

Kepler failed to load NVML on clusters using the NVIDIA GPU Operator
because the operator exposes driver libraries at /run/nvidia/driver
(driver-container root FS) but the chart mounted that hostPath directly
into the Kepler container, pulling in glibc from the driver image and
breaking dynamic linking.

Replace the direct hostPath mount with an init container that copies
only libnvidia-ml.so* into an emptyDir mounted at
/usr/local/nvidia/lib64. The emptyDir holds just the NVML libraries --
no glibc conflict.

Changes:
  manifests/helm/kepler/values.yaml
    - Chart-only knobs under daemonset.nvidia.* (not config.experimental.gpu.*
      so templates/configmap.yaml does not leak chart internals into the
      binary config). Enabling config.experimental.gpu.enabled also
      triggers the init-container plumbing.
  manifests/helm/kepler/templates/daemonset.yaml
    - Gated nvidia-libs init container, emptyDir mount, and
      LD_LIBRARY_PATH when gpu.enabled is true.
  manifests/k8s/daemonset.yaml
    - Same pattern, unconditional (raw manifest has no values system).
  .pre-commit-config.yaml
    - Local helm-lint-gpu hook for the gpu.enabled=true variant (keeps
      validation in pre-commit rather than a dedicated CI job).
  docs/user/installation.md
    - Document GPU Operator enablement, overrides, and flag coupling.

Init container notes (deep rationale):
  - Copy globs the known driver lib dirs (usr/lib64,
    usr/lib/{x86_64,aarch64}-linux-gnu) instead of running find over the
    driver root FS. That root contains the driver container's /proc:
    find walked procfs for minutes and exited nonzero on unreadable
    fdinfo entries, so init crash-looped on real GPU Operator clusters.
  - set -e with no || true on the copy pass so ENOSPC/EACCES fail init.
  - cp -P (not -L): preserve the driver symlink chain so leftover
    driver versions do not blow past the 200Mi emptyDir sizeLimit.
  - runAsUser/runAsGroup 0 with capabilities dropped for root-owned
    driver libs without depending on the image USER directive.
  - NVIDIA_VISIBLE_DEVICES / NVIDIA_MIG_MONITOR_DEVICES stay
    unconditional for the runtimeClassName: nvidia workaround.

Verified: helm lint passes both states; gpu-disabled render has no
init/nvml volume path; gpu-enabled ConfigMap keeps only
enabled/idlePower/dcgmEndpoint under experimental.gpu. Init validated
end to end on a 4-node kubeadm cluster (staged driver root with live
procfs: sub-second copy, symlink chain intact, no-driver fallback on
3 nodes, usr/lib64 layout) and against a Tesla T4 where NVML was
initialized strictly from the copied directory.

Signed-off-by: Himanshu Verma <himnshuverma10152006@gmail.com>

Author: bitflicker64

.pre-commit-config.yaml

@@ -95,3 +95,16 @@ repos:
       - id: helmlint
         name: helm-lint
         files: ^manifests/helm/[^/]+/(Chart\.yaml|values\.yaml|templates/.*)$
+
+  # GPU-enabled helm-lint: covers variants the helmlint hook above cannot
+  # run (it does not accept --set flags).
+  - repo: local
+    hooks:
+      - id: helm-lint-gpu
+        name: helm-lint (gpu enabled)
+        language: system
+        pass_filenames: false
+        files: ^manifests/helm/[^/]+/(Chart\.yaml|values\.yaml|templates/.*)$
+        entry: >
+          bash -c 'helm lint manifests/helm/kepler
+          --set config.experimental.gpu.enabled=true'

docs/user/installation.md

@@ -100,6 +100,41 @@ helm install kepler oci://quay.io/sustainable_computing_io/charts/kepler \
   --values values.yaml
 ```
 
+#### Enabling GPU Power Monitoring (NVIDIA GPU Operator)
+
+To export GPU power metrics on clusters with the [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/overview.html), enable the experimental GPU flag. The chart then adds an `nvidia-libs` init container that copies `libnvidia-ml.so*` from the host driver path into an `emptyDir`, and points `LD_LIBRARY_PATH` at it:
+
+```bash
+helm install kepler oci://quay.io/sustainable_computing_io/charts/kepler \
+  --namespace kepler \
+  --create-namespace \
+  --set config.experimental.gpu.enabled=true
+```
+
+What this changes:
+
+- Adds an `nvidia-libs` init container that copies `libnvidia-ml.so*` into a 200Mi `emptyDir` using `cp -P` (preserves the driver symlink chain; `cp -L` can exceed the size limit when leftover driver versions are present).
+- Mounts that `emptyDir` read-only at `/usr/local/nvidia/lib64` and sets `LD_LIBRARY_PATH=/usr/local/nvidia/lib64`.
+- Mounts the host driver directory (default `/run/nvidia/driver`, overridable via `daemonset.nvidia.driverPath`) with `DirectoryOrCreate` so non-GPU nodes still schedule.
+
+Defaults assume the standard GPU Operator layout. Override via `values.yaml` if needed:
+
+```yaml
+config:
+  experimental:
+    gpu:
+      enabled: true
+
+daemonset:
+  nvidia:
+    driverPath: /run/nvidia/driver
+    nvmlInitImage:
+      repository: busybox
+      tag: 1.36.1
+```
+
+Note: `config.experimental.gpu.enabled` is intentionally coupled with the chart's `daemonset.nvidia.*` plumbing. Enabling the binary flag without the init path (or the reverse) is not a supported split. Leave `enabled: false` if you do not want GPU power monitoring.
+
 #### Helm Management Commands
 
 ```bash

manifests/helm/kepler/templates/daemonset.yaml

@@ -41,6 +41,51 @@ spec:
       affinity:
         {{- toYaml . | nindent 8 }}
       {{- end }}
+      {{- if .Values.config.experimental.gpu.enabled }}
+      initContainers:
+        - name: nvidia-libs
+          image: "{{ .Values.daemonset.nvidia.nvmlInitImage.repository }}:{{ .Values.daemonset.nvidia.nvmlInitImage.tag }}"
+          imagePullPolicy: IfNotPresent
+          securityContext:
+            # Root required to read root-owned driver libs; capabilities dropped.
+            runAsNonRoot: false
+            runAsUser: 0
+            runAsGroup: 0
+            readOnlyRootFilesystem: true
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: [ALL]
+          command:
+            - sh
+            - -c
+            - |
+              set -e
+              # Glob known lib dirs only; never walk the driver rootfs — it
+              # contains the driver container's /proc (slow, unreadable).
+              # cp -P keeps driver symlink chains (avoids 200Mi emptyDir blowup).
+              copied=0
+              for d in /run-nvidia/usr/lib64 \
+                       /run-nvidia/usr/lib/x86_64-linux-gnu \
+                       /run-nvidia/usr/lib/aarch64-linux-gnu; do
+                [ -d "$d" ] || continue
+                for f in "$d"/libnvidia-ml.so*; do
+                  [ -e "$f" ] || continue
+                  cp -P "$f" /nvml-libs/
+                  copied=$((copied+1))
+                done
+              done
+              if [ "$copied" -gt 0 ]; then
+                echo "Copied $copied NVML library file(s) to /nvml-libs"
+              else
+                echo "No NVML libraries found at /run-nvidia — GPU metrics will be unavailable on this node"
+              fi
+          volumeMounts:
+            - name: lib-nvidia
+              mountPath: /run-nvidia
+              readOnly: true
+            - name: nvml-libs
+              mountPath: /nvml-libs
+      {{- end }}
       containers:
         - name: kepler
           image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
@@ -66,6 +111,11 @@ spec:
               readOnly: true
             - name: cfm
               mountPath: /etc/kepler
+            {{- if .Values.config.experimental.gpu.enabled }}
+            - name: nvml-libs
+              mountPath: /usr/local/nvidia/lib64
+              readOnly: true
+            {{- end }}
           {{- with .Values.daemonset.startupProbe }}
           startupProbe:
             {{- toYaml . | nindent 12 }}
@@ -84,11 +134,16 @@ spec:
                 fieldRef:
                   apiVersion: v1
                   fieldPath: spec.nodeName
-            # Required for NVML to see all MIG devices when running in a container
+            # Always set (independent of gpu.enabled) for runtimeClassName: nvidia.
             - name: NVIDIA_VISIBLE_DEVICES
               value: "all"
             - name: NVIDIA_MIG_MONITOR_DEVICES
               value: "all"
+            {{- if .Values.config.experimental.gpu.enabled }}
+            # Path for libnvidia-ml.so from the nvidia-libs init container.
+            - name: LD_LIBRARY_PATH
+              value: /usr/local/nvidia/lib64
+            {{- end }}
           {{- with .Values.daemonset.resources }}
           resources:
             {{- toYaml . | nindent 12 }}
@@ -103,3 +158,13 @@ spec:
         - name: cfm
           configMap:
             name: {{ include "kepler.fullname" . }}
+        {{- if .Values.config.experimental.gpu.enabled }}
+        - name: lib-nvidia
+          hostPath:
+            path: {{ .Values.daemonset.nvidia.driverPath | quote }}
+            # DirectoryOrCreate so non-GPU nodes still schedule.
+            type: DirectoryOrCreate
+        - name: nvml-libs
+          emptyDir:
+            sizeLimit: 200Mi
+        {{- end }}

manifests/helm/kepler/values.yaml

@@ -72,6 +72,20 @@ daemonset:
     initialDelaySeconds: 5
     periodSeconds: 10
 
+  # Chart-only knobs for the NVML init-container pattern. These do NOT live
+  # under .Values.config because that tree is dumped verbatim into
+  # /etc/kepler/config.yaml (templates/configmap.yaml), and the Kepler binary
+  # doesn't (and shouldn't) know about driver paths or init-image names.
+  nvidia:
+    # Path on the host where the NVIDIA GPU Operator exposes the driver
+    # container root FS. Override if your GPU Operator uses a non-default path.
+    driverPath: /run/nvidia/driver
+    # Init container used to copy NVML libraries into an emptyDir.
+    # Override repository/tag for airgapped or private-registry environments.
+    nvmlInitImage:
+      repository: busybox
+      tag: 1.36.1
+
 config:
   log:
     level: debug
@@ -129,6 +143,9 @@ config:
       zones: [] # List of zones to enable (default enable all)
       chipRules: [] # User-defined chip pairing rules (override/add to hardcoded defaults)
     gpu:
+      # Enabling this also triggers the chart's NVML init container and
+      # LD_LIBRARY_PATH injection (daemonset.nvidia.*). Leave false if you
+      # do not want GPU power monitoring.
       enabled: false # Enable experimental GPU power monitoring
       idlePower: 0 # GPU idle power in Watts (0 = auto-detect)
       dcgmEndpoint: "" # dcgm-exporter metrics URL for MIG (auto-discovered if empty)

manifests/k8s/daemonset.yaml

@@ -26,6 +26,49 @@ spec:
         - key: node-role.kubernetes.io/master
           operator: Exists
           effect: NoSchedule
+      initContainers:
+        - name: nvidia-libs
+          image: busybox:1.36.1
+          imagePullPolicy: IfNotPresent
+          securityContext:
+            # Root required to read root-owned driver libs; capabilities dropped.
+            runAsNonRoot: false
+            runAsUser: 0
+            runAsGroup: 0
+            readOnlyRootFilesystem: true
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: [ALL]
+          command:
+            - sh
+            - -c
+            - |
+              set -e
+              # Glob known lib dirs only; never walk the driver rootfs — it
+              # contains the driver container's /proc (slow, unreadable).
+              # cp -P keeps driver symlink chains (avoids 200Mi emptyDir blowup).
+              copied=0
+              for d in /run-nvidia/usr/lib64 \
+                       /run-nvidia/usr/lib/x86_64-linux-gnu \
+                       /run-nvidia/usr/lib/aarch64-linux-gnu; do
+                [ -d "$d" ] || continue
+                for f in "$d"/libnvidia-ml.so*; do
+                  [ -e "$f" ] || continue
+                  cp -P "$f" /nvml-libs/
+                  copied=$((copied+1))
+                done
+              done
+              if [ "$copied" -gt 0 ]; then
+                echo "Copied $copied NVML library file(s) to /nvml-libs"
+              else
+                echo "No NVML libraries found at /run-nvidia — GPU metrics will be unavailable on this node"
+              fi
+          volumeMounts:
+            - name: lib-nvidia
+              mountPath: /run-nvidia
+              readOnly: true
+            - name: nvml-libs
+              mountPath: /nvml-libs
       containers:
         - name: kepler
           image: <KEPLER_IMAGE>
@@ -54,8 +97,9 @@ spec:
               mountPath: /etc/kepler/config.yaml
               subPath: config.yaml
               readOnly: true
-            - name: nvidia-driver
-              mountPath: /run/nvidia/driver
+            # NVML libs only (copied by init); avoids mounting driver glibc.
+            - name: nvml-libs
+              mountPath: /usr/local/nvidia/lib64
               readOnly: true
           livenessProbe:
             httpGet:
@@ -75,8 +119,9 @@ spec:
                 fieldRef:
                   apiVersion: v1
                   fieldPath: spec.nodeName
+            # Static path; K8s cannot expand the image's existing LD_LIBRARY_PATH.
             - name: LD_LIBRARY_PATH
-              value: /run/nvidia/driver/usr/lib64
+              value: /usr/local/nvidia/lib64
             # Required for NVML to see all MIG devices when running in a container
             - name: NVIDIA_VISIBLE_DEVICES
               value: all
@@ -92,7 +137,11 @@ spec:
         - name: cfm
           configMap:
             name: kepler
-        - name: nvidia-driver
+        # lib-nvidia: host driver path; nvml-libs: init-populated emptyDir.
+        - name: lib-nvidia
           hostPath:
             path: /run/nvidia/driver
             type: DirectoryOrCreate
+        - name: nvml-libs
+          emptyDir:
+            sizeLimit: 200Mi