Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
132 changes: 117 additions & 15 deletions nemo_retriever/helm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,13 @@ nemo_retriever/helm/

## Quick start

The examples in this README use the Helm release name `retriever`.
`nemo-retriever` is the chart name. It is not the release name. When a
command omits `--namespace`, Helm installs into the current kubectl
namespace. The [Recommended minimal install](#recommended-minimal-install-26081)
and [Full teardown](#full-teardown) set `REL` and `NS` explicitly so
cleanup targets the same release.

### Persistent storage prerequisite { #persistent-storage-prerequisite }

The default chart creates **seven** PersistentVolumeClaims: three
Expand Down Expand Up @@ -458,10 +465,16 @@ Complete the [persistent storage prerequisite](#persistent-storage-prerequisite)
and the [GPU scheduling prerequisite](#gpu-scheduling-prerequisite)
before you install.

Deploy only the four core NIMs that the retriever service auto-wires (`page_elements`, `table_structure`, `ocr`, `vlm_embed`):
Deploy only the four core NIMs that the retriever service auto-wires
(`page_elements`, `table_structure`, `ocr`, `vlm_embed`). Set `REL` and
`NS` to the values you will reuse for upgrade and teardown. Replace
`NS` if you install into a namespace other than `default`.

```bash
helm install retriever ./nemo_retriever/helm \
REL=retriever
NS=default

helm install "${REL}" ./nemo_retriever/helm -n "${NS}" --create-namespace \
--set ngcImagePullSecret.create=true \
--set ngcImagePullSecret.password=$NGC_API_KEY \
--set ngcApiSecret.create=true \
Expand Down Expand Up @@ -527,29 +540,118 @@ Helm on uninstall.
| `NIMService` CR remains | **Not expected** on a normal uninstall. Usually an **orphan** from a failed install/upgrade (release never recorded the resource, or the chart renamed a NIM). |
| Deployments / GPU pods still running | Often the operator workload for a **kept** `NIMCache`, or a stale `NIMService` that Helm did not own. Check `kubectl get nimservice,nimcache -n <ns>`. |
| `nemotron-*-job-*` pods in `Error` | The NIM Operator's **model-download Job** for a `NIMCache` (not the retriever service). Failed cache pulls retry and leave Error pods until the Job or `NIMCache` is deleted. Common after a failed `helm install` when the release is rolled back but `keep` retains the cache CR. |
| `helm uninstall` appears to do nothing | Release may be missing or failed (`helm list -n <ns> -a`). CRs created before a failed install can be left without a release to clean them up. |
| `helm uninstall` appears to do nothing | Wrong release name or namespace. The chart name is `nemo-retriever`. The example release name is `retriever`. Confirm with `helm list --all-namespaces`. A missing or failed release (`helm list -n <ns> --all`) can also leave CRs without a release to clean them up. |

To change a NIM image on a later install or upgrade, delete the kept
`NIMCache` first. Refer to
[Changing a NIM image repository or tag](#changing-nim-image-repository-or-tag).

**Full teardown** (dev cluster — deletes caches and PVCs Helm kept):
### Full teardown { #full-teardown }

```bash
NS=retriever
REL=nemo-retriever
On a development cluster, remove the Helm release, kept `NIMCache`
objects, and model PVCs. Set `REL` and `NS` to the same values you used
at install time. The
[Recommended minimal install](#recommended-minimal-install-26081) uses
`REL=retriever` and `NS=default`. If you omitted `--namespace` at
install time, Helm used the current kubectl namespace. Replace `NS` if
that namespace is not `default`.

helm uninstall "${REL}" -n "${NS}" 2>/dev/null || true
1. Set the release identity and confirm the target:

# Orphans and kept NIMCaches (Helm keep does not block kubectl delete):
kubectl delete nimservice,nimcache -n "${NS}" --all
# Optional: drop model PVCs if you will re-pull from NGC
kubectl delete pvc -n "${NS}" -l 'app.kubernetes.io/managed-by=nvidia-nim-operator' 2>/dev/null || true
```
```bash
REL=retriever
NS=default

helm list -n "${NS}"
kubectl get deployment,nimservice,nimcache -n "${NS}"
```

If `helm list` shows no matching release, stop. Run
`helm list --all-namespaces` and set `REL` and `NS` to the release
you intend to remove.

2. Uninstall the release. Do not suppress Helm errors. If Helm reports
that the release was not found, correct `REL` and `NS` before you
delete NIM resources:

```bash
helm uninstall "${REL}" -n "${NS}"
helm list -n "${NS}" --all
```

3. Delete kept `NIMCache` objects and leftover `NIMService` CRs. Helm
leaves **NIMCache** objects when
`nimOperator.nimCache.keepOnUninstall` is `true` (the default). Each
NIM uses a resource name from the chart, not the Helm release name.
The following command deletes every default name this chart can
create, including optional NIMs. `--ignore-not-found` skips names
you did not install.

If you overrode `nimOperator.ocr.nimServiceName`,
`nimOperator.vlm_embed.nimServiceName`, or
`nimOperator.answer_llm.nimServiceName`, replace those three default
names with the values you set.

```bash
kubectl delete nimservice,nimcache -n "${NS}" \
nemotron-page-elements-v3 \
nemotron-table-structure-v1 \
nemotron-ocr-v2 \
llama-nemotron-embed-vl-1b-v2 \
llama-nemotron-rerank-vl-1b-v2 \
nemotron-parse \
nemotron-3-nano-omni-30b-a3b-reasoning \
audio \
answer-llm \
--ignore-not-found
```

List what remains. Delete leftover CRs by the names in that list
before you continue. Remaining objects can be renamed caches or
resources that another product owns.

```bash
kubectl get nimservice,nimcache -n "${NS}"
```

Do not run `kubectl delete nimservice,nimcache -n "${NS}" --all`
unless this namespace contains only this install. That command
deletes every `NIMService` and `NIMCache` in the namespace, including
resources that another release or product owns.

4. Optional: drop model PVCs when you re-pull weights from NGC. Confirm
claim names first. Default claim names use a `-pvc` suffix. Include
optional NIM claims. If you overrode a `nimServiceName` value, use
`<that-name>-pvc` instead of the default claim.

```bash
kubectl get pvc -n "${NS}" -l 'app.kubernetes.io/managed-by=nvidia-nim-operator'
kubectl delete pvc -n "${NS}" \
nemotron-page-elements-v3-pvc \
nemotron-table-structure-v1-pvc \
nemotron-ocr-v2-pvc \
llama-nemotron-embed-vl-1b-v2-pvc \
llama-nemotron-rerank-vl-1b-v2-pvc \
nemotron-parse-pvc \
nemotron-3-nano-omni-30b-a3b-reasoning-pvc \
audio-pvc \
answer-llm-pvc \
--ignore-not-found
```

List operator PVCs again and delete any leftover claim that belongs
to this install:

```bash
kubectl get pvc -n "${NS}" -l 'app.kubernetes.io/managed-by=nvidia-nim-operator'
```

**Dev installs** that should not retain caches on uninstall:

```bash
REL=retriever
NS=default

helm upgrade --install "${REL}" ./nemo_retriever/helm -n "${NS}" \
--set nimOperator.nimCache.keepOnUninstall=false
```
Expand Down Expand Up @@ -1169,10 +1271,10 @@ The affected NIM is unavailable while the operator re-caches weights.

1. Drain ingest traffic that depends on that NIM.
2. Confirm the live `modelPuller` value differs from the new
`repository:tag`:
`repository:tag`. Set `NS` to the namespace of your Helm release:

```bash
NS=retriever
NS=default
CACHE=nemotron-page-elements-v3

kubectl get nimcache "${CACHE}" -n "${NS}" \
Expand Down
Loading