Skip to content

Use a single shared cache for all CI workflows #1506

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Aug 4, 2025
Merged

Conversation

jakebailey
Copy link
Member

@jakebailey jakebailey commented Aug 2, 2025

This is a new attempt at getting CI caching to work, based on how I set up DT's pnpm cache, https://500.keboola.com/in-depth-of-go-build-cache/, and golang/go#58571.

It works like this:

  • The caches are produced from a dedicated create-cache workflow on main.
  • Before doing anything, the workflow sets the mtime of the files/dirs in the repo; this is important so that the build/test caches are consistent.
  • The module cache is set up with actions/cache, which restores a cache at the start, then saves a cache at the end, so any module we download will be saved.
  • We run a build, test, lint, format, which exercises the full suite of things that a workflow could do.
  • The build cache is saved.

The create-cache workflow runs on pushes to main and on a schedule early in the morning; GitHub has a 10GB limit for the Action caches, so these jobs just keep the latest caches up to date. Since the module cache rarely changes, we need the scheduled job so that we don't accidentally drop the module cache out of the rolling window.

Then, in other workflows:

  • We restore restore the mtimes so the caches will apply.
  • We restore the module and build caches from main. These are only restored, never saved.
  • The jobs proceed as normal.

This should have a few positive effects:

  • PRs have caches for everything they don't touch. Most PRs don't touch the ast or even the checker, so this should be pretty optimal.
  • Copilot jobs should have a fully populated cache from the start, without needing to run any extra setup builds/lints like we do today. We still need to run dprint as I did not add that into a cache yet (another PR). It should also be faster to add a new test, since the actual code builds would have been cached.
  • Pushes to main will reuse the cache from the previous commit, so multiple pushes to main should be no worse than if you were on a local dev machine running things in sequence.

Also:

  • Save the golangci-lint cache.

@Copilot Copilot AI review requested due to automatic review settings August 2, 2025 18:43
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements a shared CI caching strategy to improve build performance across all workflows. Instead of each job creating its own cache, a dedicated workflow pre-populates caches that all other workflows can reuse.

  • Creates a new create-cache workflow that runs on main branch pushes and scheduled daily to maintain fresh caches
  • Removes individual cache names from all CI jobs, allowing them to use the shared cache
  • Implements mtime normalization to ensure cache consistency across different runs

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
.github/workflows/create-cache.yml New workflow that creates shared caches by running build, test, lint, and format operations
.github/workflows/copilot-setup-steps.yml Removes redundant build and lint cache operations since they're now handled by shared cache
.github/workflows/ci.yml Removes individual cache-name parameters from all jobs to use shared cache instead
.github/actions/setup-go/action.yml Refactors caching logic to support both cache creation and restoration modes with mtime normalization
Comments suppressed due to low confidence (1)

.github/actions/setup-go/action.yml:37

  • The package github.com/slsyy/mtimehash/cmd/[email protected] should be verified to exist. Consider pinning to a specific commit hash or using a well-established tool for mtime management.
        find . -type f ! -path ./.git/\*\* | go run github.com/slsyy/mtimehash/cmd/[email protected] || true

@jakebailey jakebailey added this pull request to the merge queue Aug 4, 2025
Merged via the queue into main with commit f188316 Aug 4, 2025
22 checks passed
@jakebailey jakebailey deleted the jabaile/new-caching branch August 4, 2025 18:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants