perf(docker): cache image builds through cache mounts and GHA cache

[AlexTMjugador]

Overview

These changes improve the performance of our Docker image building process by enabling the reuse of cached results from previous runs in two ways:

• The Cargo target and source checkout directories are now set up as cache mounts. This allows Docker's BuildKit to reuse these directories even if the build layer cache is invalidated, which usually happens due to changes in source files copied via a previous COPY instruction.
• BuildKit caches are now stored externally as GitHub Actions cache artifacts, using a dedicated storage backend. This enables cache reuse across workflow runs.

The first improvement can significantly enhance the developer experience during local builds. The second builds upon the first to bring similar benefits to our CI environment.

That said, expectations on the CI case should be tempered. Accessing the GitHub Actions cache introduces some performance overhead, and we are already using the 10 GiB limit for cache artifacts. As a result, hit rates are expected to be low (<40%), and the overall speedup can be minimal. Still, there is no noticeable downside, and this setup offers an easy opportunity to gain speed improvements without additional compute. Combined with the improved local development experience, this seems like the right call to make.

[clrxbl]

LGTM but do wonder if we should be using registry mode caching instead as we wouldn't be running into Github storage limits this way and be able to cache more

[AlexTMjugador]

LGTM but do wonder if we should be using registry mode caching instead as we wouldn't be running into Github storage limits this way and be able to cache more

You're right that's likely to be a better cache store to use. However, as far as I can tell, it requires publishing the image and locating previously published images by their tags, which can be quite inconvenient, and that's why I initially avoided it. Still, I'll give it a try and see if we can live with those requirements.

[AlexTMjugador]

I dug a bit deeper into this, and it turns out that using pure registry caching would require publishing the cache to a separate image or tag. That feels a bit clunky, as we'd either end up exposing an internal implementation detail in our public registry, or pushing to different registries.

Instead, I've switched to using the inline cache storage backend, which stores the cache alongside the pushed image. The Docker documentation is unclear on what exactly "doesn't scale" with this backend in multi-stage builds, so I guess we'll have to try it out! The 10 GB cache GitHub offers definitely doesn't scale for monorepos anyway.