diff --git a/.github/AGENTS.md b/.github/AGENTS.md index 38d00fc8e..600d91b11 100644 --- a/.github/AGENTS.md +++ b/.github/AGENTS.md @@ -18,6 +18,7 @@ This file applies to GitHub metadata and GitHub Actions workflows. - Mobile Android CI copies `.github/gradle/gradle_plugin_repos.init.gradle` into `~/.gradle/init.d` before `flutter build apk`. Flutter's included `flutter_tools/gradle` build has no `pluginManagement` block, so a cold runner only searches the plugin portal for that classpath; the portal 303s Kotlin artifacts to Maven Central and then fails to parse `kotlin-gradle-plugins-bom`. Do not patch `$FLUTTER_ROOT` for this: the Flutter SDK cache would pick up the mutation. - Rust compiler outputs use `sccache`, installed by `.github/actions/setup-rust-sccache`. Trusted warm-cache, release, and pull-request rust jobs may connect it to a dedicated Cloudflare R2 bucket through the `SCCACHE_R2_ACCOUNT_ID` and `SCCACHE_R2_BUCKET` repository variables and the `SCCACHE_R2_ACCESS_KEY_ID` and `SCCACHE_R2_SECRET_ACCESS_KEY` repository secrets. Those credentials must have access only to the compiler-cache bucket and must never reuse the `alera-updates` bucket or its production credentials. Fork pull requests, incomplete configurations, and R2 authentication failures fall back to the runner-local cache so cache availability cannot block a build. - Cargokit sets `CARGOKIT_TEMP_DIR` under the Flutter build directory, so the fully linked `alera_native` and `code_forge` outputs under `build/**/cargokit_build` keep a separate `actions/cache` entry. Only `warm-cache.yml` may save that entry, so pull requests and release jobs restore a shared copy instead of writing private copies into their own ref scopes. Windows CI must install the pinned LunarG Vulkan SDK (`1.4.350.0`) from the official installer and verify its SHA-256. Do not use `winget`: GitHub-hosted runners fail with `APPINSTALLER_CLI_ERROR_SOURCE_DATA_MISSING` (`-1978335217`). Windows Whisper/Vulkan jobs map scratch to `R:\c` with native cargo `--target-dir` at `R:\c\n` (not `R:\c\alera_native` or `R:\alera-cargokit`: those prefixes make vulkan-shaders-gen's nested TryCompile object exceed `MAX_PATH` and `cl.exe` reports `C1083` with an empty generated-file name), set `CMAKE_GENERATOR=Ninja` so nested vulkan-shaders-gen cmake inherits Ninja, set `_CL_=/Z7 /FS` because `cl.exe` ignores `CFLAGS`, and set `GGML_CCACHE=OFF` so ggml does not wrap `cl.exe` with the rustc sccache (that pairing drops `.obj` files under Ninja). +- Steps that share nothing run concurrently with `parallel`, or with `background: true` plus a later `wait`/`wait-all`, so a job's wall time is its longest lane instead of the sum of its steps. Composite actions cannot declare either keyword, so the overlap lives in the workflows, which may still run a whole composite as a background step. A background step's outputs, `GITHUB_ENV`, and `GITHUB_PATH` changes apply only at the wait that includes it, so wait before the first step that reads them, and prefer `wait-all` when the background step has an `if:` that can skip it. Never run two Flutter or pub commands that resolve the same package concurrently: steps in a group pass `--no-pub` after one explicit resolution, because an implicit pub get rewrites `package_config.json` under the others. Never overlap two `flutter build` invocations of one project, the native integration suites, or `tool/ci/run_rust_workspace_tests.sh`, whose PTY and app-launch tests run against timing budgets. Overlap network, apt, Docker-daemon, and download work with compilation, not two compilations: a runner has 4 vCPUs (3 on macOS). Desktop Builds run 37696090551 moved the desktop-build workflow `cargo test` suites into the background beside `flutter build` with private target directories. The app build doubled on every platform, and the later debug integration builds lost the dependencies those suites leave in `rust/target`, so every leg got slower (Linux 33 to 46 min, macOS 64 to 78 min). `tune-windows-build` starts in the background before the Flutter setup and is waited on right after it. - `warm-cache.yml` exists to populate the `main` Cargokit cache scope and the shared R2 compiler cache. It must keep using the same composite actions as the pull request and release jobs, so configuration cannot drift. It must keep running the real release build, since a prologue-only warm job would leave Cargokit cold, and it must keep the scheduled/manual Linux `warm-rust` job, since that release build's rustc command line does not match PR clippy/test. `warm-rust-cache.yml` owns push warming through the same Rust checks action. Rust-only pushes must not trigger the desktop warming matrix, and `warm-cache.yml` must skip its Rust job on pushes to avoid duplicate work. - Rust checks use `ALERA_BUILD_COMMIT=unknown`; explicit nonempty commit overrides must not consult or watch Git in `alera-cli/build.rs`. Local `make rust-test` exports the stable commit and an empty `ALERA_BUILD_VERSION` only for that target and invokes Cargo directly without Bash. `tool/ci/run_rust_workspace_tests.sh` unsets `ALERA_BUILD_VERSION`; distributed builds retain their real identity. `warm-rust` uses the composite's `test-build` mode, which invokes the same two workspace test commands with `--no-run`. Keep `rust/**`, `makefile` and the Rust check scripts in the `warm-rust-cache.yml` push filters so source changes populate the cache on `main` without executing the suite again. `tool/ci/test_rust_build_cache.sh` verifies default and override identities, Windows stack flags, local checks without Bash on PATH, unaffected CLI build identities and the two runner modes without compiling the full workspace. - Every workflow must declare an explicit `permissions` block. The repository default grants the whole `GITHUB_TOKEN` scope set to jobs that only read the checkout, so a compromised action or dependency inherits write access it never needed. Workflows that build, analyze, or test declare `contents: read` at the workflow level; a job that genuinely writes something adds the single scope it needs at the job level, as `cloud-deploy.yml` and `release-cut.yml` do. CodeQL's `actions/missing-workflow-permissions` rule reports a workflow that omits it. diff --git a/.github/workflows/cloud-deploy.yml b/.github/workflows/cloud-deploy.yml index 6adc6586d..365ea905c 100644 --- a/.github/workflows/cloud-deploy.yml +++ b/.github/workflows/cloud-deploy.yml @@ -233,6 +233,37 @@ jobs: fi done + # The image build is most of this job and needs only the registry login. + # It starts once the preflight has passed and keeps building while the + # state, the current deployment and the relay state are read, then the + # plan waits for its digest. + - name: Setup Docker Buildx + if: ${{ env.DEPLOY_MODE != 'rollback' }} + uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0 + + - name: Configure Artifact Registry + if: ${{ env.DEPLOY_MODE != 'rollback' }} + run: gcloud auth configure-docker "$GCP_REGION-docker.pkg.dev" --quiet + + - name: Build And Publish Image + id: image + if: ${{ env.DEPLOY_MODE != 'rollback' }} + background: true + run: | + set -euo pipefail + repository="$GCP_REGION-docker.pkg.dev/$GCP_PROJECT_ID/alera-cloud/alera-cloud" + image_tag="$repository:$GITHUB_SHA" + docker buildx build \ + --file cloud/Dockerfile \ + --platform linux/amd64 \ + --push \ + --tag "$image_tag" \ + --metadata-file "$RUNNER_TEMP/alera-cloud-build.json" \ + cloud + digest="$(jq -er '."containerimage.digest"' "$RUNNER_TEMP/alera-cloud-build.json")" + echo "target_image=$repository@$digest" >>"$GITHUB_OUTPUT" + echo "digest=$digest" >>"$GITHUB_OUTPUT" + - name: Setup OpenTofu uses: opentofu/setup-opentofu@a1320f892987e89d278cc92dc5adc984fb93aca4 # v2.0.2 with: @@ -244,10 +275,6 @@ jobs: with: bun-version: '1.3.14' - - name: Setup Docker Buildx - if: ${{ env.DEPLOY_MODE != 'rollback' }} - uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0 - - name: Install Edge Dependencies working-directory: edge run: bun install --frozen-lockfile @@ -358,27 +385,10 @@ jobs: fi echo "state=$relay_state" >>"$GITHUB_OUTPUT" - - name: Configure Artifact Registry - if: ${{ env.DEPLOY_MODE != 'rollback' }} - run: gcloud auth configure-docker "$GCP_REGION-docker.pkg.dev" --quiet - - - name: Build And Publish Image - id: image - if: ${{ env.DEPLOY_MODE != 'rollback' }} - run: | - set -euo pipefail - repository="$GCP_REGION-docker.pkg.dev/$GCP_PROJECT_ID/alera-cloud/alera-cloud" - image_tag="$repository:$GITHUB_SHA" - docker buildx build \ - --file cloud/Dockerfile \ - --platform linux/amd64 \ - --push \ - --tag "$image_tag" \ - --metadata-file "$RUNNER_TEMP/alera-cloud-build.json" \ - cloud - digest="$(jq -er '."containerimage.digest"' "$RUNNER_TEMP/alera-cloud-build.json")" - echo "target_image=$repository@$digest" >>"$GITHUB_OUTPUT" - echo "digest=$digest" >>"$GITHUB_OUTPUT" + # A rollback builds nothing, so there is no background step to wait for + # and wait-all returns at once. + - name: Wait for image + wait-all: - name: Select Deployment Target id: target diff --git a/.github/workflows/cloud.yml b/.github/workflows/cloud.yml index bbd6c6ecd..8c6fe240a 100644 --- a/.github/workflows/cloud.yml +++ b/.github/workflows/cloud.yml @@ -53,6 +53,14 @@ jobs: with: submodules: false + # The image compiles its own release build inside the Docker daemon and + # needs nothing from the host toolchain, so it runs alongside the cargo + # checks instead of after them. It was half of this job's wall time. + - name: Build Container + id: container + background: true + run: docker build --file Dockerfile . + - name: Setup Rust uses: dtolnay/rust-toolchain@2c7215f132e9ebf062739d9130488b56d53c060c with: @@ -77,8 +85,8 @@ jobs: - name: PostgreSQL Contract Tests run: cargo test --workspace -- --ignored --test-threads=1 - - name: Build Container - run: docker build --file Dockerfile . + - name: Wait for container build + wait: container edge: runs-on: ubuntu-latest @@ -126,29 +134,33 @@ jobs: with: tofu_version: 1.12.1 - - name: Validate Production Root - working-directory: infra/production - run: | - tofu fmt -check -recursive - tofu init -backend=false -input=false - tofu validate - - - name: Validate Bootstrap Root - working-directory: infra/bootstrap/github - run: | - tofu fmt -check -recursive - tofu init -backend=false -input=false - tofu validate - - - name: Test Plan Guard - working-directory: tool/cloud - run: python3 -m unittest -v test_validate_tofu_plan.py - - - name: Validate Deployment Scripts - run: bash -n tool/cloud/deploy_worker_relay_disabled.sh tool/cloud/test_verify_production.sh tool/cloud/verify_production.sh tool/cloud/verify_worker_namespace.sh - - - name: Test Production Verifier - run: bash tool/cloud/test_verify_production.sh + # Each check reads its own root or script. tofu init downloads providers + # into each root's own .terraform directory, so the roots validate + # side by side. + - parallel: + - name: Validate Production Root + working-directory: infra/production + run: | + tofu fmt -check -recursive + tofu init -backend=false -input=false + tofu validate + + - name: Validate Bootstrap Root + working-directory: infra/bootstrap/github + run: | + tofu fmt -check -recursive + tofu init -backend=false -input=false + tofu validate + + - name: Test Plan Guard + working-directory: tool/cloud + run: python3 -m unittest -v test_validate_tofu_plan.py + + - name: Validate Deployment Scripts + run: bash -n tool/cloud/deploy_worker_relay_disabled.sh tool/cloud/test_verify_production.sh tool/cloud/verify_production.sh tool/cloud/verify_worker_namespace.sh + + - name: Test Production Verifier + run: bash tool/cloud/test_verify_production.sh cloud-ready: name: cloud-ready diff --git a/.github/workflows/desktop-build.yml b/.github/workflows/desktop-build.yml index 1add6f8c7..084211f50 100644 --- a/.github/workflows/desktop-build.yml +++ b/.github/workflows/desktop-build.yml @@ -117,15 +117,23 @@ jobs: submodules: false persist-credentials: false + # Started before the setup so the Defender exclusions already cover the + # Flutter and pub caches while the setup restores them. Its environment + # (Ninja on PATH, the R: scratch drive) applies at the wait below. + - name: Tune Windows build environment + if: runner.os == 'Windows' + background: true + uses: ./.github/actions/tune-windows-build + - name: Setup Flutter workspace uses: ./.github/actions/setup-flutter-workspace with: rust: 'true' xvfb: 'true' - - name: Tune Windows build environment - if: runner.os == 'Windows' - uses: ./.github/actions/tune-windows-build + # wait-all, not `wait: `: off Windows the tune step is skipped. + - name: Wait for Windows build environment + wait-all: - name: Verify workflow contracts and native Git integration run: >- @@ -340,13 +348,19 @@ jobs: submodules: false persist-credentials: false + # See the build job: the exclusions are in place before the cache restores. + - name: Tune Windows build environment + id: tune + background: true + uses: ./.github/actions/tune-windows-build + - name: Setup Flutter workspace uses: ./.github/actions/setup-flutter-workspace with: rust: 'true' - - name: Tune Windows build environment - uses: ./.github/actions/tune-windows-build + - name: Wait for Windows build environment + wait: tune - name: Restore cargokit build uses: actions/cache/restore@v6 diff --git a/.github/workflows/landing.yml b/.github/workflows/landing.yml index 7dd5edab2..401c3a9c6 100644 --- a/.github/workflows/landing.yml +++ b/.github/workflows/landing.yml @@ -37,6 +37,15 @@ jobs: - name: Install dependencies run: bun install --frozen-lockfile + # The browser download and its apt dependencies are the longest step + # before the end-to-end suite and touch nothing under landing/, so they + # install while the unit tests, type check and build run. Those three + # stay sequential: `astro sync` and `astro build` both write .astro/. + - name: Install Playwright Chromium + id: playwright + background: true + run: bunx playwright install --with-deps chromium + - name: Unit tests run: bun test @@ -46,8 +55,8 @@ jobs: - name: Build run: bun run build - - name: Install Playwright Chromium - run: bunx playwright install --with-deps chromium + - name: Wait for Playwright Chromium + wait: playwright - name: End-to-end tests run: bunx playwright test diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index 8c67633dd..b1a233927 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -148,20 +148,25 @@ jobs: native-assets: 'false' preflight: 'false' - - name: Format - if: needs.changes.outputs.static_dart == 'true' - run: dart format --set-exit-if-changed lib test integration_test tool - - # `dart run` would resolve the package and fire the native asset build - # hooks, which this job deliberately has no Zig for. The script imports - # only dart:io, so run the file directly. - - name: Max lines ratchet - if: needs.changes.outputs.static_dart == 'true' - run: dart tool/quality/check_max_lines.dart - - - name: Analyze - if: needs.changes.outputs.static_dart == 'true' - run: flutter analyze + # The three checks only read the resolved workspace, so they share the + # runner instead of queueing behind analyze. Analyze skips its implicit + # pub get: the setup already resolved, and a concurrent rewrite of + # package_config.json could race the other two readers. + - parallel: + - name: Format + if: needs.changes.outputs.static_dart == 'true' + run: dart format --set-exit-if-changed lib test integration_test tool + + # `dart run` would resolve the package and fire the native asset build + # hooks, which this job deliberately has no Zig for. The script imports + # only dart:io, so run the file directly. + - name: Max lines ratchet + if: needs.changes.outputs.static_dart == 'true' + run: dart tool/quality/check_max_lines.dart + + - name: Analyze + if: needs.changes.outputs.static_dart == 'true' + run: flutter analyze --no-pub packages: needs: changes @@ -181,48 +186,51 @@ jobs: native-assets: 'false' preflight: 'false' - - name: Validate shared packages - shell: bash - run: | - set -euo pipefail - for package in packages/alera_configuration; do - ( - cd "$package" - flutter pub get --enforce-lockfile - dart format --set-exit-if-changed lib test - flutter analyze --no-pub - flutter test --no-pub - ) - done - - - name: Validate vendored desktop updater - shell: bash - run: | - set -euo pipefail - ( - cd third_party/desktop_updater - # The vendored package has no committed lockfile; resolve its own - # dependency graph before running its package-local checks. - flutter pub get - flutter analyze --no-pub --no-fatal-infos - flutter test --no-pub - ) - - - name: Validate native plugin manifests - shell: bash - run: | - set -euo pipefail - for package in rust_builder mobile/rust_builder; do - ( - cd "$package" - flutter pub get - dart analyze pubspec.yaml - ) - done - - - name: Resolve standalone Dart packager - working-directory: tool/release/runtime_packager - run: dart pub get --enforce-lockfile + # Each validation resolves and checks its own package directory, so they + # run side by side. Only the packager verification needs its resolution. + - parallel: + - name: Validate shared packages + shell: bash + run: | + set -euo pipefail + for package in packages/alera_configuration; do + ( + cd "$package" + flutter pub get --enforce-lockfile + dart format --set-exit-if-changed lib test + flutter analyze --no-pub + flutter test --no-pub + ) + done + + - name: Validate vendored desktop updater + shell: bash + run: | + set -euo pipefail + ( + cd third_party/desktop_updater + # The vendored package has no committed lockfile; resolve its own + # dependency graph before running its package-local checks. + flutter pub get + flutter analyze --no-pub --no-fatal-infos + flutter test --no-pub + ) + + - name: Validate native plugin manifests + shell: bash + run: | + set -euo pipefail + for package in rust_builder mobile/rust_builder; do + ( + cd "$package" + flutter pub get + dart analyze pubspec.yaml + ) + done + + - name: Resolve standalone Dart packager + working-directory: tool/release/runtime_packager + run: dart pub get --enforce-lockfile - name: Verify standalone packaging run: dart tool/ci/verify_runtime_packager.dart @@ -243,13 +251,12 @@ jobs: with: linux-toolchain: 'false' - - name: Regenerate desktop - run: | - dart run build_runner build - dart tool/ci/normalize_generated_eof.dart - dart format lib test - + # The mobile package resolves and generates under mobile/ only, and the + # desktop pass only writes lib/ and test/ at the root, so mobile runs in + # the background behind the longer desktop pass. - name: Regenerate mobile + id: regenerate-mobile + background: true working-directory: mobile run: | flutter pub get --enforce-lockfile @@ -257,6 +264,15 @@ jobs: dart ../tool/ci/normalize_generated_eof.dart . dart format lib test + - name: Regenerate desktop + run: | + dart run build_runner build + dart tool/ci/normalize_generated_eof.dart + dart format lib test + + - name: Wait for mobile generation + wait: regenerate-mobile + - name: Require reproducible generation run: git diff --exit-code @@ -360,6 +376,14 @@ jobs: with: submodules: false + # flutter_soloud 5 compiles its native library in a build hook, and + # flutter test runs that hook for the Linux host, which needs ALSA headers. + # apt shares nothing with the Flutter setup, so it installs meanwhile. + - name: Install ALSA headers + id: alsa + background: true + run: sudo apt-get update && sudo apt-get install -y libasound2-dev + # The mobile package overrides xterm to ../third_party/xterm and runs no # native asset hooks, so it needs neither the other submodule nor Zig. - name: Setup Flutter workspace @@ -371,19 +395,20 @@ jobs: native-assets: 'false' preflight: 'false' - # flutter_soloud 5 compiles its native library in a build hook, and - # flutter test runs that hook for the Linux host, which needs ALSA headers. - - name: Install ALSA headers - run: sudo apt-get update && sudo apt-get install -y libasound2-dev + - name: Wait for ALSA headers + wait: alsa - - name: Format - run: dart format --set-exit-if-changed lib test + # Format and analyze finish inside the test run's wall time. None of + # them resolves again, so nothing rewrites package_config.json mid-run. + - parallel: + - name: Format + run: dart format --set-exit-if-changed lib test - - name: Analyze - run: flutter analyze + - name: Analyze + run: flutter analyze --no-pub - - name: Test - run: flutter test --no-pub + - name: Test + run: flutter test --no-pub # Clippy-driver and rustc do not share rustc artifacts, so this stays a # second job rather than running after rust-test. It waits for `changes` so diff --git a/.github/workflows/release-cut.yml b/.github/workflows/release-cut.yml index 1d6c14e19..7756ba3ff 100644 --- a/.github/workflows/release-cut.yml +++ b/.github/workflows/release-cut.yml @@ -354,15 +354,23 @@ jobs: ref: ${{ needs.plan.outputs.target_sha }} submodules: false + # Started before the setup so the Defender exclusions already cover the + # Flutter and pub caches while the setup restores them. Its environment + # (Ninja on PATH, the R: scratch drive) applies at the wait below. + - name: Tune Windows build environment + if: runner.os == 'Windows' + background: true + uses: ./.github/actions/tune-windows-build + - name: Setup Flutter workspace uses: ./.github/actions/setup-flutter-workspace with: rust: 'true' extra-linux-packages: rpm - - name: Tune Windows build environment - if: runner.os == 'Windows' - uses: ./.github/actions/tune-windows-build + # wait-all, not `wait: `: off Windows the tune step is skipped. + - name: Wait for Windows build environment + wait-all: - name: Restore cargokit build uses: actions/cache/restore@v6 @@ -558,56 +566,73 @@ jobs: exit 1 fi - - name: Package Windows release assets - if: matrix.platform == 'windows' - shell: pwsh - env: - RELEASE_VERSION: ${{ needs.plan.outputs.desktop_release_version }} - run: | - tar -czf "release-assets/alera-$env:RELEASE_VERSION-windows.tar.gz" -C "$env:BUNDLE_DIR" . + # Every archive reads the finished bundle and writes its own file under + # release-assets/, and each compressor is single threaded, so the + # archives of one platform are written side by side. + - parallel: + - name: Package Windows release tarball + if: matrix.platform == 'windows' + shell: pwsh + env: + RELEASE_VERSION: ${{ needs.plan.outputs.desktop_release_version }} + run: tar -czf "release-assets/alera-$env:RELEASE_VERSION-windows.tar.gz" -C "$env:BUNDLE_DIR" . + # Chocolatey and Scoop both unpack a zip natively. - Compress-Archive -Path "$env:BUNDLE_DIR\*" ` - -DestinationPath "release-assets/alera-$env:RELEASE_VERSION-windows.zip" + - name: Package Windows release zip + if: matrix.platform == 'windows' + shell: pwsh + env: + RELEASE_VERSION: ${{ needs.plan.outputs.desktop_release_version }} + run: | + Compress-Archive -Path "$env:BUNDLE_DIR\*" ` + -DestinationPath "release-assets/alera-$env:RELEASE_VERSION-windows.zip" + + - name: Package Linux release + if: matrix.platform == 'linux' + shell: bash + env: + RELEASE_VERSION: ${{ needs.plan.outputs.desktop_release_version }} + ARTIFACT_VERSION: ${{ needs.plan.outputs.desktop_artifact_version }} + BUILD_NUMBER: ${{ needs.plan.outputs.desktop_build_number }} + run: bash tool/release/package_linux.sh "$BUNDLE_DIR" release-assets "$RELEASE_VERSION" "$ARTIFACT_VERSION" "$BUILD_NUMBER" - - name: Package Linux release - if: matrix.platform == 'linux' - shell: bash - env: - RELEASE_VERSION: ${{ needs.plan.outputs.desktop_release_version }} - ARTIFACT_VERSION: ${{ needs.plan.outputs.desktop_artifact_version }} - BUILD_NUMBER: ${{ needs.plan.outputs.desktop_build_number }} - run: | - set -euo pipefail - bash tool/release/package_linux.sh "$BUNDLE_DIR" release-assets "$RELEASE_VERSION" "$ARTIFACT_VERSION" "$BUILD_NUMBER" # The unmanaged install path. Distributions outside the deb and rpm # families have no package to install, and only a directory Alera # owns can be replaced in place by the updater. - tar -czf "release-assets/alera-${RELEASE_VERSION}-linux-x64.tar.gz" \ - -C "$BUNDLE_DIR" . - - - name: Upload updater artifact - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a - with: - name: updater-${{ matrix.platform }} - path: pages/ - retention-days: 1 - if-no-files-found: error - - - name: Upload release asset artifact - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a - with: - name: release-${{ matrix.platform }} - path: release-assets/ - retention-days: 1 - if-no-files-found: error - - - name: Upload native runtime input - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a - with: - name: runtime-native-${{ matrix.platform }}-${{ matrix.native_arch }} - path: runtime-native-${{ matrix.platform }}-${{ matrix.native_arch }}.tar.gz - retention-days: 1 - if-no-files-found: error + - name: Package Linux release tarball + if: matrix.platform == 'linux' + shell: bash + env: + RELEASE_VERSION: ${{ needs.plan.outputs.desktop_release_version }} + run: | + tar -czf "release-assets/alera-${RELEASE_VERSION}-linux-x64.tar.gz" \ + -C "$BUNDLE_DIR" . + + # Three independent artifact uploads. + - parallel: + - name: Upload updater artifact + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a + with: + name: updater-${{ matrix.platform }} + path: pages/ + retention-days: 1 + if-no-files-found: error + + - name: Upload release asset artifact + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a + with: + name: release-${{ matrix.platform }} + path: release-assets/ + retention-days: 1 + if-no-files-found: error + + - name: Upload native runtime input + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a + with: + name: runtime-native-${{ matrix.platform }}-${{ matrix.native_arch }} + path: runtime-native-${{ matrix.platform }}-${{ matrix.native_arch }}.tar.gz + retention-days: 1 + if-no-files-found: error - name: Report sccache statistics if: always() @@ -881,6 +906,14 @@ jobs: ref: ${{ needs.plan.outputs.target_sha }} submodules: false + # flutter_soloud 5 compiles its native library in a build hook, and + # flutter test runs that hook for the Linux host, which needs ALSA headers. + # apt shares nothing with the Flutter setup, so it installs meanwhile. + - name: Install ALSA headers + id: alsa + background: true + run: sudo apt-get update && sudo apt-get install -y libasound2-dev + - name: Setup Flutter workspace uses: ./.github/actions/setup-flutter-workspace with: @@ -893,18 +926,24 @@ jobs: - name: Apply mobile release version locally run: dart tool/release/update_mobile_pubspec_version.dart "${{ needs.plan.outputs.mobile_artifact_version }}" "${{ needs.plan.outputs.mobile_build_number }}" - - name: Analyze mobile app + # The version bump touches pubspec.yaml, so resolve once here. Analyze + # and test then skip their implicit pub get and cannot race each other + # rewriting package_config.json. + - name: Resolve mobile dependencies working-directory: mobile - run: flutter analyze + run: flutter pub get - # flutter_soloud 5 compiles its native library in a build hook, and - # flutter test runs that hook for the Linux host, which needs ALSA headers. - - name: Install ALSA headers - run: sudo apt-get update && sudo apt-get install -y libasound2-dev + - name: Wait for ALSA headers + wait: alsa - - name: Test mobile app - working-directory: mobile - run: flutter test + - parallel: + - name: Analyze mobile app + working-directory: mobile + run: flutter analyze --no-pub + + - name: Test mobile app + working-directory: mobile + run: flutter test --no-pub - name: Decode Android signing keystore shell: bash @@ -1043,21 +1082,13 @@ jobs: - name: Setup Dart uses: dart-lang/setup-dart@65eb853c7ba17dde3be364c3d2858773e7144260 - - name: Setup Flutter workspace - if: needs.plan.outputs.desktop_should_release == 'true' - uses: ./.github/actions/setup-flutter-workspace - with: - linux-toolchain: 'false' - native-assets: 'false' - preflight: 'false' - - - name: Configure git author - run: | - git config user.name "github-actions[bot]" - git config user.email "41898282+github-actions[bot]@users.noreply.github.com" - + # The downloads and tool installs below write only their own directories + # and need nothing from the Flutter setup, so they run in the background + # while it restores the SDK. Their outputs and PATH changes apply at the + # wait before the first step that reads them. - name: Download updater artifacts if: needs.plan.outputs.desktop_should_release == 'true' + background: true uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c with: pattern: updater-* @@ -1066,6 +1097,7 @@ jobs: - name: Download release assets if: needs.plan.outputs.desktop_should_release == 'true' + background: true uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c with: pattern: release-* @@ -1074,11 +1106,46 @@ jobs: - name: Download Android release assets if: needs.plan.outputs.mobile_should_release == 'true' + background: true uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c with: name: mobile-android-release-assets path: mobile-release-assets + - name: Install Linux repository tools + if: needs.plan.outputs.desktop_should_release == 'true' && needs.plan.outputs.channel == 'stable' + background: true + run: sudo apt-get update && sudo apt-get install -y apt-utils createrepo-c + + - name: Ensure AWS CLI + if: needs.plan.outputs.desktop_should_release == 'true' + background: true + run: | + set -euo pipefail + if ! command -v aws >/dev/null 2>&1; then + python3 -m pip install --user awscli + echo "$HOME/.local/bin" >>"$GITHUB_PATH" + export PATH="$HOME/.local/bin:$PATH" + fi + aws --version + + - name: Setup Flutter workspace + if: needs.plan.outputs.desktop_should_release == 'true' + uses: ./.github/actions/setup-flutter-workspace + with: + linux-toolchain: 'false' + native-assets: 'false' + preflight: 'false' + + - name: Configure git author + run: | + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + + # wait-all, not `wait: `: which of them run depends on the plan. + - name: Wait for downloads and tools + wait-all: + - name: Verify complete mobile release if: needs.plan.outputs.mobile_should_release == 'true' shell: bash @@ -1101,10 +1168,6 @@ jobs: fi (cd mobile-release-assets && sha256sum -c ./*.sha256) - - name: Install Linux repository tools - if: needs.plan.outputs.desktop_should_release == 'true' && needs.plan.outputs.channel == 'stable' - run: sudo apt-get update && sudo apt-get install -y apt-utils createrepo-c - - name: Merge and verify desktop update channel if: needs.plan.outputs.desktop_should_release == 'true' env: @@ -1154,17 +1217,6 @@ jobs: ALERA_LINUX_GPG_KEY_ID: ${{ secrets.ALERA_LINUX_GPG_KEY_ID }} run: bash tool/release/build_linux_repositories.sh public release-assets - - name: Ensure AWS CLI - if: needs.plan.outputs.desktop_should_release == 'true' - run: | - set -euo pipefail - if ! command -v aws >/dev/null 2>&1; then - python3 -m pip install --user awscli - echo "$HOME/.local/bin" >>"$GITHUB_PATH" - export PATH="$HOME/.local/bin:$PATH" - fi - aws --version - - name: Verify R2 deployment prerequisites if: needs.plan.outputs.desktop_should_release == 'true' env: @@ -1329,44 +1381,48 @@ jobs: --notes-file mobile-release-notes.md fi - - name: Upload desktop release assets - if: needs.plan.outputs.desktop_should_release == 'true' - env: - GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} - TAG: ${{ needs.plan.outputs.desktop_tag }} - run: | - bash tool/release/upload_release_assets.sh \ - "$TAG" \ - "$GITHUB_REPOSITORY" \ - release-assets/* \ - "public/$ALERA_ARCHIVE_FILE" \ - "public/$ALERA_RUNTIME_ARCHIVE_FILE" - - - name: Upload mobile release assets - if: needs.plan.outputs.mobile_should_release == 'true' - env: - GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} - TAG: ${{ needs.plan.outputs.mobile_tag }} - run: | - bash tool/release/upload_release_assets.sh \ - "$TAG" \ - "$GITHUB_REPOSITORY" \ - mobile-release-assets/* - - - name: Attest desktop release assets - if: needs.plan.outputs.desktop_should_release == 'true' - uses: actions/attest-build-provenance@e8998f949152b193b063cb0ec769d69d929409be - with: - subject-path: | - release-assets/* - public/${{ env.ALERA_ARCHIVE_FILE }} - public/${{ env.ALERA_RUNTIME_ARCHIVE_FILE }} - - - name: Attest mobile release assets - if: needs.plan.outputs.mobile_should_release == 'true' - uses: actions/attest-build-provenance@e8998f949152b193b063cb0ec769d69d929409be - with: - subject-path: mobile-release-assets/* + # Both releases are still drafts here. The uploads go to different + # releases and the attestations hash the local files, so all four run + # at once; the public flips below still wait for every one of them. + - parallel: + - name: Upload desktop release assets + if: needs.plan.outputs.desktop_should_release == 'true' + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + TAG: ${{ needs.plan.outputs.desktop_tag }} + run: | + bash tool/release/upload_release_assets.sh \ + "$TAG" \ + "$GITHUB_REPOSITORY" \ + release-assets/* \ + "public/$ALERA_ARCHIVE_FILE" \ + "public/$ALERA_RUNTIME_ARCHIVE_FILE" + + - name: Upload mobile release assets + if: needs.plan.outputs.mobile_should_release == 'true' + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + TAG: ${{ needs.plan.outputs.mobile_tag }} + run: | + bash tool/release/upload_release_assets.sh \ + "$TAG" \ + "$GITHUB_REPOSITORY" \ + mobile-release-assets/* + + - name: Attest desktop release assets + if: needs.plan.outputs.desktop_should_release == 'true' + uses: actions/attest-build-provenance@e8998f949152b193b063cb0ec769d69d929409be + with: + subject-path: | + release-assets/* + public/${{ env.ALERA_ARCHIVE_FILE }} + public/${{ env.ALERA_RUNTIME_ARCHIVE_FILE }} + + - name: Attest mobile release assets + if: needs.plan.outputs.mobile_should_release == 'true' + uses: actions/attest-build-provenance@e8998f949152b193b063cb0ec769d69d929409be + with: + subject-path: mobile-release-assets/* - name: Publish mobile release if: needs.plan.outputs.mobile_should_release == 'true' diff --git a/.github/workflows/warm-cache.yml b/.github/workflows/warm-cache.yml index 1320c3b6d..c6493c1bd 100644 --- a/.github/workflows/warm-cache.yml +++ b/.github/workflows/warm-cache.yml @@ -59,14 +59,22 @@ jobs: with: submodules: false + # Started before the setup so the Defender exclusions already cover the + # Flutter and pub caches while the setup restores them. Its environment + # (Ninja on PATH, the R: scratch drive) applies at the wait below. + - name: Tune Windows build environment + if: runner.os == 'Windows' + background: true + uses: ./.github/actions/tune-windows-build + - name: Setup Flutter workspace uses: ./.github/actions/setup-flutter-workspace with: rust: 'true' - - name: Tune Windows build environment - if: runner.os == 'Windows' - uses: ./.github/actions/tune-windows-build + # wait-all, not `wait: `: off Windows the tune step is skipped. + - name: Wait for Windows build environment + wait-all: - name: Restore cargokit build uses: actions/cache/restore@v6 diff --git a/.github/workflows/windows-pty-job.yml b/.github/workflows/windows-pty-job.yml index 4e71b21af..2fd04066f 100644 --- a/.github/workflows/windows-pty-job.yml +++ b/.github/workflows/windows-pty-job.yml @@ -35,6 +35,14 @@ jobs: with: submodules: false + # Started before the setup so the Defender exclusions already cover the + # Flutter and pub caches while the setup restores them. Its environment + # (Ninja on PATH, the R: scratch drive) applies at the wait below. + - name: Tune Windows build environment + id: tune + background: true + uses: ./.github/actions/tune-windows-build + - name: Setup Flutter workspace uses: ./.github/actions/setup-flutter-workspace with: @@ -43,8 +51,8 @@ jobs: preflight: 'false' rust: 'true' - - name: Tune Windows build environment - uses: ./.github/actions/tune-windows-build + - name: Wait for Windows build environment + wait: tune - name: Test Windows PTY Job lifecycle run: >- diff --git a/test/unit/desktop_validation_workflow_test.dart b/test/unit/desktop_validation_workflow_test.dart index 0318cdce5..0f1345f59 100644 --- a/test/unit/desktop_validation_workflow_test.dart +++ b/test/unit/desktop_validation_workflow_test.dart @@ -158,6 +158,35 @@ void main() { expect(workflows['run'], contains('-p alera-core --features runtime')); expect(workflows['run'], contains('--lib workflow_')); expect(workflows['run'], contains('--locked')); + // Run 37696090551 overlapped both suites with the app build: the build + // doubled on every platform and the debug integration builds lost the + // dependencies these suites leave in rust/target, a net loss of ~13 min + // on Linux and ~14 min on macOS. + for (final name in [ + 'Verify workflow contracts and native Git integration', + 'Verify native workflow launch and recovery', + 'Verify Windows workflow launch and recovery', + ]) { + expect(step('build', name)['background'], isNull, reason: name); + } + }); + + test('Windows tuning runs in the background ahead of the setup', () { + for (final job in ['build', 'windows_native_smoke']) { + final jobSteps = steps(job); + final names = [for (final s in jobSteps) s['name']]; + final tune = names.indexOf('Tune Windows build environment'); + final setup = names.indexOf('Setup Flutter workspace'); + expect(jobSteps[tune]['background'], isTrue, reason: job); + expect(tune, lessThan(setup), reason: job); + // Its PATH and env changes only apply once a wait includes it. + final wait = jobSteps[setup + 1]; + expect( + wait.containsKey('wait') || wait.containsKey('wait-all'), + isTrue, + reason: job, + ); + } }); test( diff --git a/test/unit/pr_checks_workflow_test.dart b/test/unit/pr_checks_workflow_test.dart index bb44baf2d..05bff26cc 100644 --- a/test/unit/pr_checks_workflow_test.dart +++ b/test/unit/pr_checks_workflow_test.dart @@ -146,4 +146,28 @@ void main() { final pushChecks = rust['jobs']['warm-rust']['strategy']['matrix']; expect(pushChecks, scheduledChecks); }); + + test('mobile checks wait for ALSA before the parallel test run', () { + final pr = loadYaml( + File('.github/workflows/pr.yml').readAsStringSync(), + ) as YamlMap; + final steps = (pr['jobs']['mobile']['steps'] as YamlList).cast(); + final alsa = steps.indexWhere((s) => s['name'] == 'Install ALSA headers'); + final setup = steps.indexWhere( + (s) => s['name'] == 'Setup Flutter workspace', + ); + final wait = steps.indexWhere((s) => s['wait'] == steps[alsa]['id']); + final group = steps.indexWhere((s) => s.containsKey('parallel')); + expect(steps[alsa]['background'], isTrue); + expect(alsa, lessThan(setup)); + expect(wait, greaterThan(setup)); + expect(group, greaterThan(wait)); + final checks = (steps[group]['parallel'] as YamlList).cast(); + expect([for (final s in checks) s['name']], ['Format', 'Analyze', 'Test']); + // A concurrent implicit pub get would rewrite package_config.json while + // the other checks read it. + for (final check in checks.where((s) => s['name'] != 'Format')) { + expect(check['run'], contains('--no-pub')); + } + }); }