[Bug] 403 Access Denied when pulling self-hosted NVCF Helm charts from NGC OCI Registry

[logn-xu]

Bug Description

During the local self-hosted NVCF installation setup via k3d, the helmfile apply (or nvcf-cli self-hosted up) command fails with a 403 Forbidden: Access Denied error when trying to pull the dependency Helm charts (such as helm-nvcf-cassandra, helm-nvcf-openbao-server, etc.) from the nvcr.io OCI registry.

Environment

• OS/Distribution: Fedora
• Tools Version: helmfile (latest/v1.x), helm (v4/v3.x), nvcf-cli (built from main branch)
• Cluster Layout: Local single-cluster k3d (ncp-local)

Steps to Reproduce

1. Followed the Quickstart guide, configured /etc/hosts and exported registry credentials.
2. Logged into the Helm registry successfully via:

   helm registry login nvcr.io -u '$oauthtoken' -p "${NGC_API_KEY}"```
   

3. Modified the deploy/stacks/self-managed/environments/base.yaml to use our actual NGC Org and Team slugs (replacing YOUR_ORG/YOUR_TEAM placeholders with valid, lowercase-compliant paths).
4. Ran the installation command or invoked the underlying helmfile directly:

HELMFILE_ENV=local helmfile -f deploy/stacks/self-managed/helmfile.d/ --sequential-helmfiles --track-mode helm-legacy apply```



## Expected Behavior

Helm should successfully pull the OCI artifacts from `nvcr.io` using the authenticated session credentials.

## Actual Logs / Output

```text
Pulling nvcr.io/logn_club/test-logn/helm-nvcf-cert-manager:0.1.0
Pulling nvcr.io/logn_club/test-logn/helm-nvcf-openbao-server:0.30.23
Pulling nvcr.io/logn_club/test-logn/helm-nvcf-cassandra:0.15.5
Pulling nvcr.io/logn_club/test-logn/helm-nvcf-nats:0.7.1
in /home/logn/[github.com/nvidia/nvcf/deploy/stacks/self-managed/helmfile.d/01-dependencies.yaml.gotmpl](https://github.com/nvidia/nvcf/deploy/stacks/self-managed/helmfile.d/01-dependencies.yaml.gotmpl): [release "cassandra": command "/usr/bin/helm" exited with non-zero status:

PATH:
  /usr/bin/helm

ARGS:
  0: helm
  1: pull
  2: oci://nvcr.io/logn_club/test-logn/helm-nvcf-cassandra
  3: --destination
  4: /home/logn/.cache/helmfile/nvcf/helm-nvcf-cassandra/0.15.5
  5: --untar
  6: --version
  7: 0.15.5
  8: --devel

STDERR:
  Error: failed to perform "FetchReference" on source: GET "[https://nvcr.io/v2/logn_club/test-logn/helm-nvcf-cassandra/manifests/0.15.5](https://nvcr.io/v2/logn_club/test-logn/helm-nvcf-cassandra/manifests/0.15.5)": GET "[https://nvcr.io/proxy_auth?scope=repository%3Alogn_club%2Ftest-logn%2Fhelm-nvcf-cassandra%3Apull](https://nvcr.io/proxy_auth?scope=repository%3Alogn_club%2Ftest-logn%2Fhelm-nvcf-cassandra%3Apull)": response status code 403: denied: Access Denied

Additional Context / Questions

• Manually testing helm pull oci://nvcr.io/nvidia/nvcf/helm-nvcf-openbao-server --version 0.30.23 also yields a 403 Access Denied error.
• Are these NVCF Helm charts hosted under a different public/private global organization on NGC that our organization needs explicit entitlement for? Or is there a specific predefined registry path that we shouldn't overwrite with our own Org/Team names?

[charles-hsiao]

Hi @logn-xu, I hit the same wall, dug through the repo, and verified the behavior live with an NGC API key from a regular NCA org. Short version: the Helmfile stack is designed to pull NVCF charts from an OCI registry that you mirror them into first, and the quickstart never mentions that prerequisite.

Root cause

1. The NVCF Helm charts are not OCI artifacts on nvcr.io, so the OCI path 403s no matter what credentials you use.

The helmfile declares the chart source as an OCI repository built from global.helm.sources (helmfile.d/01-dependencies.yaml.gotmpl). But per the warning in docs/user/image-mirroring.md, NVCF charts in the nvidia/nvcf namespace are NGC Helm repository charts served from https://helm.ngc.nvidia.com/nvidia/nvcf; they are not published as OCI artifacts. I confirmed this experimentally: with a valid key and a successful helm registry login nvcr.io, helm pull oci://nvcr.io/nvidia/nvcf/helm-nvcf-openbao-server --version 0.30.23 still returns the exact proxy_auth ... 403: denied: Access Denied you saw, while the same key pulls the same charts fine from the NGC Helm repository.

2. YOUR_ORG/YOUR_TEAM in base.yaml means a registry you populate, not a path NVIDIA populates for you.

The "Prepare Helm charts for Helmfile" section of docs/user/manifest.md documents exactly this: copy each required chart version into the OCI registry configured by global.helm.sources before deploying. docs/user/helmfile-installation.md says the same: "helm charts must be configured to a user-managed registry". So nvcr.io/logn_club/test-logn 403s simply because nothing has been pushed there yet.

One extra wrinkle: the NGC Helm repository itself is not anonymously readable. https://helm.ngc.nvidia.com/nvidia/nvcf/index.yaml returns 403 without credentials (unlike nvidia and nvidia/omniverse, which are anonymous), so the unauthenticated helm repo add examples currently in image-mirroring.md fail as written. With an NGC API key from an ordinary NCA org (not the nvcf-onprem org mentioned in the mirroring prerequisites) both the Helm repository and the nvcr.io/nvidia/nvcf container images pull fine.

Workaround: mirror the charts, then re-run the install

# 1. Add the NVCF Helm repo. Credentials are required; anonymous access 403s.
helm repo add nvcf https://helm.ngc.nvidia.com/nvidia/nvcf \
  --username '$oauthtoken' --password "${NGC_API_KEY}" --force-update
helm repo update

# 2. Pull each chart listed in the artifact manifest
#    (prerelease chart versions need --devel)
helm pull nvcf/helm-nvcf-cassandra --version 0.15.5

# 3. Push into the registry path you set in base.yaml
helm registry login nvcr.io -u '$oauthtoken' -p "${NGC_API_KEY}"
helm push helm-nvcf-cassandra-0.15.5.tgz oci://nvcr.io/logn_club/test-logn

Repeat for every chart in the artifact manifest; its "Prepare Helm charts for Helmfile" section is the canonical procedure. Container images are simpler: nvcr.io/nvidia/nvcf/<image> pulls directly with an authenticated key, so for global.image you can either mirror or point at nvidia/nvcf with imagePullSecrets configured.

Suggested docs fix

The underlying gap: quickstart.md has users run helm registry login nvcr.io and lists "An NGC API key with access to the NVCF chart and image registry" as a prerequisite, but never mentions chart mirroring, and helm.ngc.nvidia.com appears nowhere in it. Anyone following the quickstart top to bottom hits this 403. I would be happy to open a docs PR that:

• adds a "mirror NVCF charts" prerequisite step to quickstart.md, cross-linking image-mirroring.md and manifest.md
• clarifies the YOUR_ORG/YOUR_TEAM comment in base.yaml to state it must be a registry you have mirrored the charts into
• adds credentials to the helm repo add examples in image-mirroring.md, which currently fail anonymously

@sbaum1994 two questions to confirm the intended workflow:

1. Should https://helm.ngc.nvidia.com/nvidia/nvcf be anonymously readable like nvidia/omniverse, or is authenticated access the intended state? The docs call these charts public but the index 403s without a key.
2. Is there a plan to publish the charts as OCI artifacts on nvcr.io so the quickstart works as written, or should the quickstart document the mirror step?

[logn-xu]

---

Update: I managed to resolve the Helm chart pulling issue by switching from the OCI registry reference to the classic Helm repo instead — i.e. using https://helm.ngc.nvidia.com/nvidia/nvcf as the repo URL rather than oci://nvcr.io/.... That got past the 403 Access Denied error on the Helm charts. Thanks for the pointers!

However, I've now hit a new issue: some of the container images fail to pull. Specifically:

nvcr.io/nvidia/nvcf/nvcf-openbao:2.5.4-nv-1.3.0
spec.containers{openbao}: Failed to pull image "nvcr.io/nvidia/nvcf/nvcf-openbao:2.5.4-nv-1.3.0":
failed to pull and unpack image "nvcr.io/nvidia/nvcf/nvcf-openbao:2.5.4-nv-1.3.0":
failed to resolve reference "nvcr.io/nvidia/nvcf/nvcf-openbao:2.5.4-nv-1.3.0":
failed to authorize: failed to fetch anonymous token: unexpected status from GET request
to https://nvcr.io/proxy_auth?scope=repository%3Anvidia%2Fnvcf%2Fnvcf-openbao%3Apull: 403 Forbidden
  Warning  Failed     46m (x3 over 47m)      kubelet            spec.containers{openbao}: Error: ErrImagePull

This looks like the same kind of access/entitlement issue as before, but now on the container image side (nvcf-openbao) rather than the Helm chart. Is this image also gated behind a specific org/team entitlement, or is there a different registry path/pull secret configuration expected for these container images?

---