Skip to content

Commit 81c9a84

Browse files
coreyjadamscoreyjadams
committed
Update contracts in github nightly uv pipeline to ensure we build and retrieve caches
correctly. At least, that's the *intention*.
1 parent 3dce279 commit 81c9a84

6 files changed

Lines changed: 539 additions & 94 deletions

File tree

‎.github/CACHE_CONTRACT.md‎

Lines changed: 175 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,175 @@
1+
# Nightly UV Cache Contract
2+
3+
This document is the authoritative reference for how the
4+
`Nightly Github UV Workflow`
5+
([.github/workflows/github-nightly-uv.yml](workflows/github-nightly-uv.yml))
6+
publishes Python environment caches and how downstream PR workflows must
7+
consume them. PR gating relies on these contracts being honored on both
8+
the producer and consumer side; do not weaken them without updating this
9+
document.
10+
11+
## Two caches, two contracts
12+
13+
The pipeline maintains two strictly disjoint caches with different
14+
invalidation rules. Conflating them is the historical bug class this
15+
design exists to forbid.
16+
17+
### A. uv download cache (`~/.cache/uv`)
18+
19+
| Property | Value |
20+
|---|---|
21+
| Key | `<UV_CACHE_KEY_PREFIX>-latest` |
22+
| Prefix encodes | container image + Python version + uv version |
23+
| Suffix | literal `latest` (mutable slot, refreshed via delete-before-save) |
24+
| Contents | every wheel uv has ever downloaded for this baseline; additive across lockfile changes |
25+
| Invalidates when | container image, CUDA version, Python version, or uv version changes (prefix change → new slot) |
26+
| Does **not** invalidate on | `uv.lock` or `pyproject.toml` changes |
27+
| Restore semantics | **fail-open**; missing cache only costs download time, never correctness |
28+
| Save semantics | always save (delete the existing entry first, then save, then verify with `gh cache list`) |
29+
30+
The uv download cache is purely a speed optimisation. Anything that
31+
correctness depends on must come from the venv cache.
32+
33+
### B. venv cache (`.venv`)
34+
35+
| Property | Value |
36+
|---|---|
37+
| Key | `<VENV_CACHE_KEY_PREFIX>-<lockhash>` |
38+
| Prefix encodes | container image + Python version + uv version + extras tag (e.g. `cu12`) |
39+
| Suffix | `hashFiles('uv.lock', 'pyproject.toml')`, computed once per job and propagated via `needs.<job>.outputs.lockhash` |
40+
| Contents | the fully realized `.venv` produced by `uv sync --frozen --group dev --extra <tag>` against the committed lockfile |
41+
| Invalidates when | any prefix component changes, or the lockfile hash changes |
42+
| Restore semantics | **exact-match only, no `restore-keys` fallback** |
43+
| Save semantics | standard `actions/cache/save`. Same lockhash → same content → save no-ops, which is correct |
44+
45+
The extras tag (`cu12`, `cu13`, ...) is part of the prefix so cu12 and
46+
cu13 builds never overwrite each other.
47+
48+
The lockhash includes both `uv.lock` and `pyproject.toml`. If a PR
49+
touches `pyproject.toml` without regenerating `uv.lock`, the build fails
50+
loudly during `uv sync --frozen` rather than silently producing a
51+
mismatched venv.
52+
53+
## PR consumer contracts
54+
55+
PR workflows that gate on the nightly venv MUST implement one of two
56+
exhaustive paths.
57+
58+
### Contract 1 — PR does not touch `pyproject.toml` or `uv.lock`
59+
60+
The PR's lockhash equals the hash that the most recent successful nightly
61+
saved under.
62+
63+
```yaml
64+
- name: Restore uv download cache (fail-open)
65+
uses: actions/cache/restore@v4
66+
with:
67+
path: ~/.cache/uv
68+
key: <UV_CACHE_KEY_PREFIX>-latest
69+
# NO fail-on-cache-miss. Missing uv cache is acceptable here.
70+
71+
- name: Restore venv cache (exact match, MUST hit)
72+
uses: actions/cache/restore@v4
73+
with:
74+
path: .venv
75+
key: <VENV_CACHE_KEY_PREFIX>-${{ hashFiles('uv.lock', 'pyproject.toml') }}
76+
fail-on-cache-miss: true
77+
# NO restore-keys. A partial match would silently degrade test
78+
# validity.
79+
80+
- name: Use the env, read-only
81+
env:
82+
UV_FROZEN: "1"
83+
UV_NO_SYNC: "1"
84+
run: |
85+
.venv/bin/python -c "import torch; print(torch.__version__)"
86+
uv run --no-sync python -m pytest ...
87+
```
88+
89+
Guarantees:
90+
91+
- Either the venv is byte-identical to what nightly validated, or the
92+
job fails on cache miss.
93+
- `UV_FROZEN=1` and `UV_NO_SYNC=1` (plus the `--no-sync` flags) make it
94+
impossible for any subsequent `uv run` to mutate the restored venv.
95+
- `physicsnemo` itself is installed editable, so the PR's source code
96+
changes are picked up without rebuilding the venv.
97+
98+
### Contract 2 — PR updates `pyproject.toml` and/or `uv.lock`
99+
100+
The PR's lockhash is new; the venv cache misses by design.
101+
102+
```yaml
103+
- name: Restore uv download cache (fail-open)
104+
uses: actions/cache/restore@v4
105+
with:
106+
path: ~/.cache/uv
107+
key: <UV_CACHE_KEY_PREFIX>-latest
108+
109+
- name: Restore venv cache (will miss; that is fine)
110+
id: venv-restore
111+
uses: actions/cache/restore@v4
112+
with:
113+
path: .venv
114+
key: <VENV_CACHE_KEY_PREFIX>-${{ hashFiles('uv.lock', 'pyproject.toml') }}
115+
# No fail-on-cache-miss; we expect to miss on lock-change PRs.
116+
117+
- name: Clean-build venv
118+
if: steps.venv-restore.outputs.cache-hit != 'true'
119+
env:
120+
UV_LINK_MODE: copy
121+
UV_FROZEN: "1"
122+
UV_NO_SYNC: "1"
123+
run: |
124+
rm -rf .venv
125+
uv sync --frozen --group dev --extra cu12
126+
# Optional: assert the lockfile was not mutated, e.g. with sha256sum
127+
# before/after. setup-uv-env does this automatically.
128+
```
129+
130+
Guarantees:
131+
132+
- `rm -rf .venv` ensures no leftover state from a previous PR push or a
133+
partial restore-keys hit. (Restore-keys is forbidden anyway, but this
134+
is cheap insurance.)
135+
- `--frozen` + `UV_FROZEN=1` ensure the resolver cannot rewrite
136+
`uv.lock` in CI. If the PR shipped a stale lock, the job fails fast
137+
with a clear error rather than papering over the mismatch.
138+
- The uv download cache (restored fail-open) supplies most wheels, so
139+
the rebuild is fast even though the venv itself is fresh.
140+
141+
## Operational notes
142+
143+
- **Concurrency**: the nightly workflow declares
144+
`concurrency: nightly-github-uv` with `cancel-in-progress: false` so
145+
two overlapping runs cannot race on the static `-latest` uv cache key.
146+
- **Save verification**: after `actions/cache/save@v4` writes the uv
147+
download cache slot, the workflow re-queries `gh cache list` to
148+
confirm the entry exists. `cache/save` silently no-ops on key
149+
collision; without verification a corrupted slot can persist for days.
150+
- **Lockfile-mutation guard**: [.github/actions/setup-uv-env/action.yml](actions/setup-uv-env/action.yml)
151+
snapshots `sha256(uv.lock)` and `sha256(pyproject.toml)` before any uv
152+
command runs and compares them again at the end. Any drift (caused by
153+
a forgotten `--frozen`, a dropped `--extra`, etc.) trips this guard
154+
and fails the job with a pointed error message.
155+
- **uv version pin**: `bootstrap-cudnn-ci` installs a pinned uv version
156+
via `https://astral.sh/uv/<version>/install.sh` and asserts the
157+
installed binary matches. The pin is what allows the uv version to
158+
appear in the cache key prefix without surprise invalidations.
159+
160+
## Bumping any of the baseline values
161+
162+
If you change the container image, CUDA version, Python version, uv
163+
version, or extras tag, you must update both:
164+
165+
1. The matching `env:` value at the top of
166+
[.github/workflows/github-nightly-uv.yml](workflows/github-nightly-uv.yml).
167+
2. The corresponding literal embedded in `UV_CACHE_KEY_PREFIX` and
168+
`VENV_CACHE_KEY_PREFIX` (GitHub Actions does not support env-to-env
169+
references within the same `env:` block, so these are kept in
170+
lockstep manually).
171+
172+
The first nightly run after a baseline bump will miss both caches, do a
173+
full rebuild, and republish under the new prefix. Existing PR workflows
174+
that pin to the old prefix will hard-fail until they are updated, which
175+
is the desired behaviour.

‎.github/actions/bootstrap-cudnn-ci/action.yml‎

Lines changed: 25 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,14 @@ inputs:
55
description: Python major.minor expected in the container
66
required: false
77
default: "3.12"
8+
uv-version:
9+
description: |
10+
Exact uv version to install. Pinning is required because the uv
11+
version is part of the cache key prefix (a surprise uv upgrade would
12+
otherwise silently invalidate the wheel store). Bump in lockstep
13+
with the workflow's UV_VERSION env value.
14+
required: false
15+
default: "0.11.7"
816
runs:
917
using: composite
1018
steps:
@@ -30,19 +38,33 @@ runs:
3038
ln -sf /usr/bin/python3 /usr/bin/python
3139
rm -rf /var/lib/apt/lists/*
3240
33-
- name: Install uv
41+
- name: Install uv (pinned)
3442
shell: bash
43+
env:
44+
UV_VERSION: ${{ inputs.uv-version }}
3545
run: |
3646
set -euo pipefail
37-
curl -LsSf https://astral.sh/uv/install.sh | sh
38-
echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
47+
# Pinned installer URL: https://astral.sh/uv/<version>/install.sh
48+
# ensures the installed binary version matches the cache-key tag.
49+
curl -LsSf "https://astral.sh/uv/${UV_VERSION}/install.sh" | sh
50+
# Modern uv (>= 0.5) installs to ~/.local/bin, not ~/.cargo/bin.
51+
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
3952
4053
- name: Print toolchain versions
4154
shell: bash
55+
env:
56+
EXPECTED_UV_VERSION: ${{ inputs.uv-version }}
4257
run: |
4358
set -euo pipefail
4459
python3 --version
4560
uv --version
61+
# Hard-fail if the installed uv does not match the pin -- otherwise
62+
# the cache prefix would lie about which uv produced the wheels.
63+
actual_uv="$(uv --version | awk '{print $2}')"
64+
if [ "$actual_uv" != "$EXPECTED_UV_VERSION" ]; then
65+
echo "::error::uv version mismatch: expected ${EXPECTED_UV_VERSION}, got ${actual_uv}"
66+
exit 1
67+
fi
4668
gcc --version | head -n 1
4769
cmake --version | head -n 1
4870
if command -v nvcc >/dev/null 2>&1; then

0 commit comments

Comments
 (0)