Skip to content

Commit d56865f

Browse files
marcpaternomarcpaterno
committed
Factorize CI across multiple branches
Previously, ci.yml held all job definitions and only triggered on pushes to master. The nightly cron also ran only on master. The 300-line ci.yml is split into three workflow files. ci-reusable.yml holds all three CI pipeline stages and is triggered only via workflow_call, never directly by GitHub events. ci.yml becomes a thin caller that triggers on push to any branch and on pull_request; a gateway job (should-run) reads .github/ci-branches.json at runtime and skips the pipeline for pushes to branches not in the supported list. Pull requests always run. nightly.yml takes over the cron schedule and uses a load-branches job to build a matrix from the same JSON file, calling ci-reusable.yml once per supported branch. .github/ci-branches.json is the single source of truth for which branches receive CI on push and are tested nightly. Adding or retiring a branch requires editing only that file; no workflow file needs to change. CONTRIBUTING.md gains a CI System Architecture section explaining the three-file structure, the gateway mechanism, and the one-file rule for branch maintenance. README.md gains a nightly CI status badge and a Contributing section linking to CONTRIBUTING.md.
1 parent d3a8b87 commit d56865f

File tree

6 files changed

+451
-280
lines changed

6 files changed

+451
-280
lines changed

‎.github/ci-branches.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
["master", "v1.14"]

‎.github/workflows/ci-reusable.yml‎

Lines changed: 307 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,307 @@
1+
# Reusable CI Workflow
2+
# ====================
3+
# This workflow contains the full CI pipeline for Firecrown.
4+
# It is called by ci.yml (for push/PR events) and nightly.yml
5+
# (for scheduled multi-branch testing). All job definitions
6+
# live here so that adding a new supported branch never requires
7+
# duplicating job steps.
8+
#
9+
# STAGE 1: QUICK VALIDATION (Parallel with Stage 2)
10+
# - Goal: 2-3 minute feedback on common PR issues.
11+
# - Actions: Linting, formatting, and Jupyter Notebook cleanliness checks.
12+
# - Optimization: Uses environment caching to minimize setup time.
13+
#
14+
# STAGE 2: FULL COMPATIBILITY MATRIX
15+
# - Goal: Verify Firecrown across all supported environments.
16+
# - Actions: Comprehensive unit and integration testing.
17+
# - Matrix: Covers OS (Linux/macOS) and Python versions defined in the job below.
18+
# - Optimization: Ubuntu + Python 3.12 performs coverage analysis.
19+
#
20+
# STAGE 3: DOWNSTREAM & DOCUMENTATION (Sequential - depends on Stage 2)
21+
# - Goal: Verify ecosystem integration and build stability.
22+
# - Actions: Tests with Smokescreen and Augur; full documentation verification.
23+
# - Optimization: Only runs if the main matrix succeeds to save resources.
24+
#
25+
# MAINTENANCE NOTE:
26+
# When updating supported Python versions, only the matrix configuration
27+
# in Stage 2 needs to be changed. This header is intentionally abstract.
28+
#
29+
# MacOS Note:
30+
# Stage 2 includes a workaround for RPATH issues in 'isitgr' on macOS to
31+
# ensure libraries are correctly loaded during integration tests.
32+
33+
name: firecrown-ci-reusable
34+
on:
35+
workflow_call:
36+
inputs:
37+
ref:
38+
description: >
39+
Git ref (branch name, tag, or SHA) to check out and test.
40+
Defaults to the ref that triggered the calling workflow.
41+
required: false
42+
type: string
43+
default: ""
44+
secrets:
45+
CODECOV_TOKEN:
46+
description: "Codecov upload token"
47+
required: false
48+
49+
env:
50+
CONDA_ENV: firecrown_developer
51+
CACHE_VERSION: 15
52+
53+
jobs:
54+
quick-validation:
55+
name: Quick validation
56+
runs-on: ubuntu-latest
57+
steps:
58+
- uses: actions/checkout@v5
59+
with:
60+
ref: ${{ inputs.ref }}
61+
- name: Setting up Miniforge
62+
uses: conda-incubator/setup-miniconda@v3
63+
with:
64+
miniforge-version: latest
65+
python-version: "3.12"
66+
activate-environment: ${{ env.CONDA_ENV }}
67+
- name: Cache date
68+
id: get-date
69+
run: echo "today=$(/bin/date -u '+%Y%m%d')" >> $GITHUB_OUTPUT
70+
shell: bash
71+
- name: Compute environment.yml hash
72+
id: env-hash
73+
run: |
74+
if command -v sha256sum &>/dev/null 2>&1; then
75+
h=$(sha256sum environment.yml | cut -d' ' -f1)
76+
else
77+
h=$(shasum -a 256 environment.yml | cut -d' ' -f1)
78+
fi
79+
echo "env_hash=${h}" >> $GITHUB_OUTPUT
80+
shell: bash
81+
- name: Cache Conda env
82+
uses: actions/cache/restore@v4
83+
id: cache
84+
with:
85+
path: ${{ env.CONDA }}/envs
86+
key: conda-Linux-X64-py3.12-${{ steps.get-date.outputs.today }}-${{ steps.env-hash.outputs.env_hash }}-v${{ env.CACHE_VERSION }}
87+
- name: Update environment
88+
if: steps.cache.outputs.cache-hit != 'true'
89+
run: |
90+
python .github/update_ci.py 3.12
91+
conda env update -n ${{ env.CONDA_ENV }} -f env_tmp.yml --prune
92+
- name: Save conda environment
93+
if: steps.cache.outputs.cache-hit != 'true'
94+
uses: actions/cache/save@v4
95+
with:
96+
path: ${{ env.CONDA }}/envs
97+
key: ${{ steps.cache.outputs.cache-primary-key }}
98+
- name: Quick setup and linting
99+
shell: bash -l {0}
100+
run: |
101+
pip install --no-deps -e .
102+
make lint
103+
- name: Ensure clear Jupyter Notebooks
104+
uses: ResearchSoftwareActions/EnsureCleanNotebooksAction@1.1
105+
106+
firecrown-miniforge:
107+
name: (${{ matrix.os }}, py${{ matrix.python-version }})${{ matrix.coverage && ' + coverage' || '' }}
108+
runs-on: ${{ matrix.os }}-latest
109+
strategy:
110+
fail-fast: false
111+
matrix:
112+
os: ["ubuntu", "macos"]
113+
python-version: ["3.11", "3.12", "3.13", "3.14"]
114+
include:
115+
- os: ubuntu
116+
python-version: "3.12"
117+
coverage: true
118+
steps:
119+
- uses: actions/checkout@v5
120+
with:
121+
ref: ${{ inputs.ref }}
122+
- name: Setting up Miniforge
123+
uses: conda-incubator/setup-miniconda@v3
124+
with:
125+
miniforge-version: latest
126+
python-version: ${{ matrix.python-version }}
127+
activate-environment: ${{ env.CONDA_ENV }}
128+
- name: Cache date
129+
id: get-date
130+
run: echo "today=$(/bin/date -u '+%Y%m%d')" >> $GITHUB_OUTPUT
131+
shell: bash
132+
- name: Compute environment.yml hash
133+
id: env-hash
134+
run: |
135+
if command -v sha256sum &>/dev/null 2>&1; then
136+
h=$(sha256sum environment.yml | cut -d' ' -f1)
137+
else
138+
h=$(shasum -a 256 environment.yml | cut -d' ' -f1)
139+
fi
140+
echo "env_hash=${h}" >> $GITHUB_OUTPUT
141+
shell: bash
142+
- name: Cache Conda env
143+
uses: actions/cache/restore@v4
144+
id: cache
145+
with:
146+
path: ${{ env.CONDA }}/envs
147+
key: conda-${{ runner.os }}-${{ runner.arch }}-py${{ matrix.python-version }}-${{ steps.get-date.outputs.today }}-${{ steps.env-hash.outputs.env_hash }}-v${{ env.CACHE_VERSION }}
148+
- name: Update environment
149+
if: steps.cache.outputs.cache-hit != 'true'
150+
run: |
151+
python .github/update_ci.py ${{ matrix.python-version }}
152+
conda env update -n ${{ env.CONDA_ENV }} -f env_tmp.yml --prune
153+
- name: Setting up CosmoSIS and Cobaya
154+
if: steps.cache.outputs.cache-hit != 'true'
155+
shell: bash -l {0}
156+
run: |
157+
# Sequential setup of dependencies to avoid environment corruption
158+
source ${CONDA_PREFIX}/bin/cosmosis-configure
159+
pushd ${CONDA_PREFIX}
160+
cosmosis-build-standard-library
161+
export CSL_DIR=${PWD}/cosmosis-standard-library
162+
conda env config vars set CSL_DIR=${CSL_DIR}
163+
popd
164+
python -m pip install cobaya --no-deps
165+
- name: Remove redundant rpath and import isitgr
166+
if: ${{ (matrix.os == 'macos') && (steps.cache.outputs.cache-hit != 'true') }}
167+
shell: bash -l {0}
168+
run: |
169+
install_name_tool -delete_rpath /Users/runner/miniconda3/envs/firecrown_developer/lib /Users/runner/miniconda3/envs/firecrown_developer/lib/python3.1?/site-packages/isitgr/isitgrlib.so || true
170+
python -c "import isitgr"
171+
- name: Save conda-forge, CosmoSIS and Cobaya environment
172+
id: cache-primes-save
173+
uses: actions/cache/save@v4
174+
with:
175+
path: ${{ env.CONDA }}/envs
176+
key: ${{ steps.cache.outputs.cache-primary-key }}
177+
- name: Setting up Firecrown
178+
shell: bash -l {0}
179+
run: |
180+
export FIRECROWN_DIR=${PWD}
181+
conda env config vars set FIRECROWN_DIR=${FIRECROWN_DIR}
182+
pip install --no-deps -e .
183+
conda list
184+
- name: Code quality checks
185+
shell: bash -l {0}
186+
run: make lint
187+
- name: Run CI tests
188+
shell: bash -l {0}
189+
run: |
190+
if [[ "${{ matrix.coverage || false }}" == "true" ]]; then
191+
make test-ci
192+
else
193+
make test-all
194+
fi
195+
- name: Upload coverage reports to Codecov
196+
if: ${{ matrix.coverage == true }}
197+
uses: codecov/codecov-action@v5
198+
with:
199+
files: ./coverage.xml
200+
fail_ci_if_error: true
201+
verbose: true
202+
use_oidc: false
203+
# TODO: Remove the following two lines when GPG verification is fixed
204+
disable_search: true
205+
disable_file_fixes: true
206+
env:
207+
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
208+
# TODO: Remove when GPG key verification issue is resolved upstream
209+
CODECOV_SKIP_GPG_VERIFY: true
210+
211+
external-dependencies:
212+
name: External dependencies and documentation
213+
needs: [firecrown-miniforge]
214+
runs-on: macos-latest
215+
steps:
216+
- uses: actions/checkout@v5
217+
with:
218+
ref: ${{ inputs.ref }}
219+
- name: Setting up Miniforge
220+
uses: conda-incubator/setup-miniconda@v3
221+
with:
222+
miniforge-version: latest
223+
python-version: "3.13"
224+
activate-environment: ${{ env.CONDA_ENV }}
225+
- name: Cache date
226+
id: get-date
227+
run: echo "today=$(/bin/date -u '+%Y%m%d')" >> $GITHUB_OUTPUT
228+
shell: bash
229+
- name: Compute environment.yml hash
230+
id: env-hash
231+
run: |
232+
if command -v sha256sum &>/dev/null 2>&1; then
233+
h=$(sha256sum environment.yml | cut -d' ' -f1)
234+
else
235+
h=$(shasum -a 256 environment.yml | cut -d' ' -f1)
236+
fi
237+
echo "env_hash=${h}" >> $GITHUB_OUTPUT
238+
shell: bash
239+
- name: Cache Conda env
240+
uses: actions/cache/restore@v4
241+
id: cache
242+
with:
243+
path: ${{ env.CONDA }}/envs
244+
key: conda-${{ runner.os }}-${{ runner.arch }}-py3.13-${{ steps.get-date.outputs.today }}-${{ steps.env-hash.outputs.env_hash }}-v${{ env.CACHE_VERSION }}
245+
- name: Update environment
246+
if: steps.cache.outputs.cache-hit != 'true'
247+
run: |
248+
python .github/update_ci.py 3.13
249+
conda env update -n ${{ env.CONDA_ENV }} -f env_tmp.yml --prune
250+
- name: Remove redundant rpath and import isitgr
251+
shell: bash -l {0}
252+
run: |
253+
install_name_tool -delete_rpath /Users/runner/miniconda3/envs/firecrown_developer/lib /Users/runner/miniconda3/envs/firecrown_developer/lib/python3.1?/site-packages/isitgr/isitgrlib.so || true
254+
python -c "import isitgr"
255+
- name: Setting up CosmoSIS and Cobaya
256+
if: steps.cache.outputs.cache-hit != 'true'
257+
shell: bash -l {0}
258+
run: |
259+
# Sequential setup of dependencies to avoid environment corruption
260+
source ${CONDA_PREFIX}/bin/cosmosis-configure
261+
pushd ${CONDA_PREFIX}
262+
cosmosis-build-standard-library
263+
export CSL_DIR=${PWD}/cosmosis-standard-library
264+
conda env config vars set CSL_DIR=${CSL_DIR}
265+
popd
266+
python -m pip install cobaya --no-deps
267+
- name: Save conda-forge, CosmoSIS and Cobaya environment
268+
id: cache-primes-save
269+
uses: actions/cache/save@v4
270+
with:
271+
path: ${{ env.CONDA }}/envs
272+
key: ${{ steps.cache.outputs.cache-primary-key }}
273+
- name: Create firecrown_developer environment
274+
shell: bash -l {0}
275+
run: |
276+
export FIRECROWN_DIR=${PWD}
277+
conda env config vars set FIRECROWN_DIR=${FIRECROWN_DIR}
278+
pip install --no-deps -e .
279+
conda list
280+
- name: Cloning Smokescreen
281+
uses: actions/checkout@v5
282+
with:
283+
repository: "marcpaterno/smokescreen"
284+
path: "smokescreen"
285+
ref: "adjust-to-firecrown-pr-586"
286+
- name: Cloning Augur
287+
uses: actions/checkout@v5
288+
with:
289+
repository: "lsstdesc/augur"
290+
path: "augur"
291+
ref: "b59cfaf3dec90aa606a4add453a5a77e0c8ea942"
292+
- name: Build and verify tutorials and documentation
293+
shell: bash -l {0}
294+
run: make docs-verify
295+
- name: Pip-install Augur and test it
296+
shell: bash -l {0}
297+
run: |
298+
conda install jinja2 tjpcov
299+
cd augur
300+
pip install -e .
301+
python -m pytest .
302+
- name: Pip-install Smokescreen and test it
303+
shell: bash -l {0}
304+
run: |
305+
cd smokescreen
306+
pip install -e .
307+
python -m pytest tests

0 commit comments

Comments
 (0)