Skip to content

cache-integration.md

Latest commit

 

History

History
353 lines (267 loc) · 12.6 KB

cache-integration.md

File metadata and controls

353 lines (267 loc) · 12.6 KB
Raw

Cache Integration

How GloriousFlywheel runners consume Attic and Bazel today, and where that differs from the advertised enrollment front door.

This page documents the current internal live contract, not historical cache examples.

Bazel Remote Cache

Current internal compatibility path

The historical runner cache endpoint is still served and injected for internal runner compatibility:

grpc://bazel-cache.nix-cache.svc.cluster.local:9092

That endpoint is not the product enrollment front door, is not reachable from ordinary off-cluster machines, and must not be copied into consumer repos. Some supported self-hosted runners still inject BAZEL_REMOTE_CACHE automatically so existing CI can keep using cache-backed local execution while the token-enforced GF path graduates.

Recommended .bazelrc

build:runner-pool --remote_upload_local_results=true
build:runner-pool --remote_timeout=60

Bazel rc files do not expand shell environment variables in option values. Pass --remote_cache="$BAZEL_REMOTE_CACHE" from a wrapper, Just recipe, or workflow command after the strict cache-attachment preflight succeeds.

Consumer REAPI lace-up

Spoke repositories should copy the canonical consumer wrapper from examples/bazel/gloriousflywheel-bazel.sh rather than inventing endpoint logic. The wrapper has two modes:

  • GF_BAZEL_SUBSTRATE_MODE=shared-cache-backed requires BAZEL_REMOTE_CACHE and selects --config=ci-cached
  • GF_BAZEL_SUBSTRATE_MODE=executor-backed requires both BAZEL_REMOTE_CACHE and BAZEL_REMOTE_EXECUTOR, selects --config=executor-backed, passes --remote_executor, forces remote strategy, and disables local fallback

Auth stays runtime-injected through BAZEL_CREDENTIAL_HELPER, BAZEL_REMOTE_HEADER, BAZEL_REMOTE_CACHE_HEADER, and BAZEL_REMOTE_EXEC_HEADER. Do not commit endpoint values, bearer headers, or credential-helper paths into a downstream .bazelrc.

Fleet-managed developer machines should get these values from the Fleet Profile Distribution module path. That path installs non-secret endpoint/profile metadata and exposes the machine-readable GF_FLYWHEEL_PROFILE_STATE; it does not mint or store bearer tokens.

To materialize a non-secret env file for a consumer repo from this repo:

just flywheel-consumer-env shared-cache-backed \
  --cache-endpoint "$BAZEL_REMOTE_CACHE" \
  --credential-helper "gf-reapi-cell.example.internal=/run/current-system/sw/bin/gf-reapi-credhelper" \
  --credential-helper-token-file "$XDG_RUNTIME_DIR/gloriousflywheel/gf-reapi-token.jwt" \
  --write .env.flywheel.local
source .env.flywheel.local

Use cluster-executor only from an in-cluster runner/operator shell, and use port-forward-cache or port-forward-executor only for bounded local proofs. The wrapper rejects localhost cache or executor endpoints unless the env file also sets GF_BAZEL_LOCAL_PROOF=port-forward.

Current boundary

  • shared self-hosted runners remain supported Bazel-cache compatibility consumers
  • new enrollment should prefer the token-enforced GF path below instead of teaching the historical :9092 cache as product surface
  • local dev or external-consumer Bazel addressing is not yet a stable platform promise unless a profile carries a real routable endpoint and runtime auth
  • do not onboard new users against ad hoc or historical external Bazel hostnames

Token-Enforced Front Door

TIN-2161 adds the first token-enforced routable cache front door contract for gf-reapi-cell as a REAPI cache service. This is separate from the historical cluster-internal bazel-cache.nix-cache.svc.cluster.local runner path. A routable shared-cache-backed or executor-backed profile must carry runtime auth wiring, normally:

export BAZEL_CREDENTIAL_HELPER=gf-reapi-cell.example.internal=/run/current-system/sw/bin/gf-reapi-credhelper
export GF_REAPI_CREDENTIAL_HELPER_TOKEN_FILE=/run/user/1000/gloriousflywheel/gf-reapi-token.jwt

flywheel-doctor and flywheel-verify reject routable cache/executor endpoints that omit runtime auth wiring. flywheel-consumer-env can write the non-secret helper path and token-file path, but it must not write bearer token material. The server-side gate is TestAuthzEnforcedCacheFrontDoorScopes; the repo-level gate is just gf-reapi-cache-frontdoor-contract-check.

This is still a cache-front-door gate, not broad/default RBE or durable CAS/AC authority. TIN-2162 has now proved the shared-runner / in-cluster CI audience: a fresh tinyland-inc/site.scaffold@main checkout used only the advertised flywheel-enroll / flywheel-verify path, rejected a forced-remote no-auth probe, and completed //:build with nonzero remote actions. Off-cluster laptop ingress remains a separate product gate.

Developer-machine attachment

Developer machines use the same variables as CI, but they do not auto-discover the in-cluster gRPC service. The expected local sequence is:

direnv allow
just info
just flywheel-doctor

If just info reports compatibility-local-only, do not run heavy Bazel work as if it were on the shared substrate. Set BAZEL_REMOTE_CACHE only from a real routable endpoint supplied by endpoint/profile authority. The preferred fleet path is the NixOS or Home Manager profile module; the local fallback can be rendered with:

just flywheel-consumer-env shared-cache-backed \
  --cache-endpoint "$BAZEL_REMOTE_CACHE" \
  --credential-helper "gf-reapi-cell.example.internal=/run/current-system/sw/bin/gf-reapi-credhelper" \
  --credential-helper-token-file "$XDG_RUNTIME_DIR/gloriousflywheel/gf-reapi-token.jwt" \
  --write .env.flywheel.local

After the profile is loaded, run:

just flywheel-verify

flywheel-verify validates the local profile state and fails closed for unattached or contradictory environments. It does not prove cache hits or mint tokens; run the bounded attachment proof after it passes.

The TIN-758 policy decision remains no ad hoc public cache hostname: tailnet-routable or public endpoints require a separate infrastructure/auth decision before they are advertised as default product behavior. Then use:

just cache-contract-strict
just developer-cache-attachment-proof

Internal operators can make the in-cluster service routable to a developer machine with a bounded local port-forward:

kubectl --context honey -n nix-cache port-forward svc/bazel-cache 19092:9092

Then, in the developer shell:

export BAZEL_REMOTE_CACHE=grpc://127.0.0.1:19092
export GF_BAZEL_SUBSTRATE_MODE=shared-cache-backed
export GF_BAZEL_LOCAL_PROOF=port-forward
just cache-contract-strict
just developer-cache-attachment-proof //:deployment_bundle false

The equivalent profile command is:

just flywheel-consumer-env port-forward-cache --write .env.flywheel.local
source .env.flywheel.local

just dev-attach is the current alpha helper for this path. It prints the same contract state, refuses stale endpoints and executor-backed env as proof of local cache attachment, refuses localhost endpoints unless the bounded proof marker is present, verifies that Nix is attached through NIX_CONFIG, and can run the bounded proof with:

just dev-attach --proof

The proof command defaults to //:deployment_bundle and read-only remote-cache use. Trusted operators can opt into cache writes explicitly:

just developer-cache-attachment-proof //:deployment_bundle true

Just recipe arguments are positional here. Pass the target before true; just developer-cache-attachment-proof true treats true as the Bazel target.

Use the broader all-target wrapper only after the bounded proof is green:

just bazel-build-cached

This keeps the product contract honest: local dev and CI share the same cache contract when the endpoint is present, but current main does not promise a stable public or general-consumer external Bazel hostname.

The 2026-04-29 internal proof used the Honey port-forward path above, reached the cache service, and built //:deployment_bundle in read-only cache mode. That proves developer-machine shared-cache attachment for an operator-provided endpoint. It still does not prove Bazel remote execution or a public Bazel cache hostname.

If BAZEL_REMOTE_CACHE is unset, just developer-cache-attachment-proof must fail at just cache-contract-strict. That is the expected pre-attachment state, not a Bazel failure.

Attic Nix Cache

Stable runner default

Nix-capable self-hosted runners use the cluster-internal Attic API endpoint:

http://attic.nix-cache.svc.cluster.local

They also receive the default shared cache name:

ATTIC_CACHE=main

Human-facing read path

For internal human-operated machines or downstream consumers that only need read access, use the HTTPS cache path:

https://nix-cache.tinyland.dev/main

If you are using the attic CLI itself against the API base, use:

https://nix-cache.tinyland.dev

Local Nix configuration

extra-substituters = https://nix-cache.tinyland.dev/main
extra-trusted-public-keys = main:YOUR_PUBLIC_KEY_HERE

Inside this repo, .envrc derives ATTIC_PUBLIC_KEY from the committed live runner tfvars when the variable is not already set. That keeps local Nix, ARC, and GitLab compatibility runners on the same public trust key without copying it into a private .env file. Use just attic-public-key-contract-check to verify the committed runner tfvars still agree.

Use just cache-contract-nix-strict inside the dev shell to verify that the local shell is not only carrying Attic variables, but has attached Nix itself: NIX_CONFIG must contain both the configured Attic substituter and the public trust key.

Current boundary

  • self-hosted runners get Attic configuration automatically
  • internal users may read from the HTTPS cache path when the shared main cache is public-read and ATTIC_PUBLIC_KEY is trusted locally
  • Attic writes remain internal and credentialed
  • FlakeHub, not Attic, is the publication/discovery surface

Trusted prewarm

For advanced KVM closures, use the trusted operator prewarm contract in KVM Cache Prewarm. PR jobs remain read-only for Attic writes; ATTIC_TOKEN belongs only in protected default-branch, scheduled, or manual operator publication contexts.

GitHub Actions Usage

Recommended current pattern:

jobs:
  build:
    runs-on: tinyland-nix
    steps:
      - uses: actions/checkout@v6
      - uses: tinyland-inc/GloriousFlywheel/.github/actions/nix-job@main
        with:
          command: nix build .#default

For existing internal compatibility workloads, add your repo-local wrapper or .bazelrc config and let the runner-provided BAZEL_REMOTE_CACHE value drive the explicit --remote_cache flag. For new enrollment and fresh-repo proofs, use the packaged front-door kit plus flywheel-github-oidc-profile so the cache/executor path is token-backed instead of a copied cluster DNS literal.

ARC runner lanes can optionally inject BAZEL_REMOTE_EXECUTOR from the backend-neutral bazel_executor_endpoint module input. That wiring is empty by default. If it is enabled, the lane also keeps BAZEL_REMOTE_CACHE, sets GF_BAZEL_SUBSTRATE_MODE=executor-backed, and carries GF_BAZEL_REMOTE_EXECUTION_PLATFORM. This is endpoint plumbing for the repo-managed wrapper; target eligibility still comes from the checked RBE target manifest.

GitLab Compatibility Usage

The GitLab path still exists, but it is compatibility-only.

Current compatibility truth:

  • Nix-capable GitLab runners inject ATTIC_SERVER and ATTIC_CACHE
  • supported GitLab runners may inject BAZEL_REMOTE_CACHE
  • these values should point at the same live cache family used by the shared platform, not old attic-cache-dev examples

Explicitly Stale Values

Do not use these as current guidance:

  • grpc://bazel-cache.attic-cache-dev.svc.cluster.local:9092
  • https://attic.dev-cluster.example.com
  • https://attic.tinyland.dev
  • old fuzzy-dev cache hostnames

Troubleshooting Hints

  • if BAZEL_REMOTE_CACHE is empty on CI, you are not on the expected self-hosted runner path
  • if BAZEL_REMOTE_CACHE is empty on a developer machine, you are in the intentional compatibility-local-only mode until an operator-provided endpoint is set
  • if ATTIC_SERVER is empty on a Nix-capable runner, check the runner stack inputs and runtime env injection
  • if a self-hosted Bazel or Nix job works only when hard-coding a legacy hostname, the repo is still depending on a stale contract and should be rewritten

Related