Skip to content

ci: run independent workflow steps in parallel - #923

Merged
leynier merged 2 commits into
mainfrom
perf/parallel-ci-steps
Oct 8, 2026
Merged

leynier merged 2 commits into
mainfrom
perf/parallel-ci-steps

Conversation

@leynier

@leynier leynier commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner

Summary

Uses the new GitHub Actions parallel / background / wait / wait-all step keywords to overlap steps that share nothing. Composite actions cannot declare them, so all the overlap lives in the workflows. Estimates come from step timings of recent runs (PR Checks 37542513947, Cut Release 37544847437, Cloud Deploy 37571177618, Desktop Builds 36038731698).

Workflow / job Change Expected saving
cloud backend docker build in the background during fmt/clippy/test measured: 387 s → 228 s
cloud-deploy deploy Image build/push starts after the preflight and runs while state and current deployment are read ~40 s
pr mobile ALSA apt during setup; format, analyze, test in parallel ~40 s
pr generation Mobile regeneration in the background behind desktop measured: 323 s → 262 s
pr static, packages Independent checks in parallel ~10–20 s each
landing Playwright install during unit tests, type check and build ~15 s
cloud infrastructure Both tofu roots and script checks in parallel ~10 s
release-cut Android: ALSA during setup, analyze ‖ test. Desktop: archives and artifact uploads in parallel. Publish: downloads, apt and AWS CLI during the Flutter setup; uploads and attestations in parallel ~30–60 s per job
Windows jobs tune-windows-build in the background before the setup, so the Defender exclusions are in place while the Flutter and pub caches are restored not measured

Tried and reverted: running the desktop-build workflow cargo test suites in the background beside flutter build (run 37696090551). The build doubled on every platform and the debug integration builds lost the shared rust/target dependencies (Linux 33 → 46 min, macOS 64 → 78 min). A contract test now keeps them in the foreground.

Not changed: rust test/clippy (one composite step, and the PTY suites flake under concurrency), PR test shards (setup then test), iOS device ‖ simulator builds (they share Generated.xcconfig), mobile-build Android, Windows PTY tests (the same target dir serializes them).

.github/AGENTS.md documents the rules: wait before reading a background step's env/outputs, prefer wait-all for conditional steps, --no-pub inside groups, one target dir per background cargo invocation, no overlap of builds/native suites.

Validation

  • New contract tests: the desktop workflow suites stay in the foreground; tuning runs before the setup with a wait after it; mobile waits for ALSA before the parallel group and uses --no-pub. Each fails when its change is reverted.
  • Existing workflow contract tests, max-lines ratchet, git diff --check, analyze.
  • actionlint 1.7.12 does not support the new keywords yet. GitHub runs on this PR passed: PR Checks, Cloud, Landing, Windows PTY, and a Cut Release dry_run dispatch, which parses release-cut.yml and only plans. On the Desktop Builds dispatch, Linux and macOS passed. Windows failed in a later debug cmake_install (INSTALL.vcxproj, MSB3073) after the background steps had finished, and the two previous dispatches also failed on build windows.

Risks

  • A failing background step only fails the job at its wait, so its failure is reported later than before.
  • The Windows tuning in the background had no measurable effect on setup time (variance is ±200 s). It is kept because it is harmless, not because it is faster.
  • release-cut publish and cloud-deploy only run on a real release or deploy. The release-cut file is parsed on this PR. cloud-deploy is not, and it uses the same keywords as the jobs that run here.

Uses the GitHub Actions parallel and background step keywords to overlap steps that share nothing: the desktop-build workflow tests with the app build (private cargo target dirs so they never take the build's lock), the cloud container build with the cargo checks and with the production state reads, apt and Playwright installs with the Flutter or Bun setup, the release downloads with the publish setup, and the Windows tuning with the cache restores so the Defender exclusions apply to them.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ One overlap miss on Linux and macOS. The rest matches the background / wait-all rules, and the jobs that already ran on this branch succeeded.

Reviewed changes

Reviewed the parallel-step conversion across the PR, desktop, release, cloud, landing, warm-cache, and Windows PTY workflows, plus the contract tests that pin ordering and cargo target dirs.

  • Desktop build overlap. Workflow cargo tests run beside flutter build, with a wait-all before the native timing suites. Windows tuning starts before setup so Ninja and ALERA_CARGOKIT_TEMP_DIR apply at the wait.
  • PR Checks. Static, package, generation, and mobile steps that do not share a package resolution now run together, with --no-pub after one resolve and ALSA waited on before mobile tests.
  • Release cut. Archives, artifact uploads, publish-time downloads, and draft uploads plus attestations overlap. Conditional background steps use wait-all.
  • Cloud and landing. The backend image build overlaps cargo checks, deploy pushes the image while state is read, infra checks run side by side, and Playwright installs during the landing unit tests and build.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using grok-4.7 | 𝕏

Comment thread .github/workflows/desktop-build.yml Outdated
Desktop Builds run 37696090551 overlapped them with flutter build: the build doubled on every platform and the debug integration builds lost the rust/target dependencies those suites leave behind, so each leg got slower.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Reviewed 450e31ea, which puts the desktop-build workflow cargo tests back in the foreground after run 37696090551 showed the overlap was a net loss.

  • Foreground workflow tests. Restored the three desktop-build cargo suites to sequential steps ahead of flutter build, matching main, so later debug integration builds still see the artifacts those suites leave in rust/target.
  • Regression guard. Asserted those steps are not background, and recorded the measured slowdown in .github/AGENTS.md.

Pullfrog  | View workflow run | Using grok-4.7 | 𝕏

@leynier
leynier merged commit 51edc40 into main Oct 8, 2026
27 checks passed
@leynier
leynier deleted the perf/parallel-ci-steps branch October 8, 2026 00:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant