feat: Add reproducible builds release workflows and push images to DockerHub #7614
Conversation
|
|
|
Ubuntu seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
MoeMahhouk
force-pushed
the
unstable
branch
2 times, most recently
from
82dcfef to
238fbaa
Compare
|
Doing some testing on this, will post the comment when ready |
|
Some required checks have failed. Could you please take a look @MoeMahhouk? 🙏 |
mergify
bot
added
waiting-on-author
Thanks for reviewing it. |
Is there a reason why is this PR still a draft? |
Not really, I am waiting for your final feedback to open it for review/merge. |
There was a problem hiding this comment.
I have tested this and it is working. Would be great to get some comments from the devs
I have added a label test-reproducible, that will trigger the reproducible builds CI to run. When we have release PR, we can add this label
to run the check and ensure that the binaries are reproducible.
|
@chong-he , I just wanted to double check if this is waiting on me for anything else so far? |
No The PR is now waiting for other team members to review |
|
Hi @MoeMahhouk, this pull request has been closed automatically due to 30 days of inactivity. If you’d like to continue working on it, feel free to reopen at any time. |
|
Re-opening this PR as it was awaiting review. |
jimmygchen
removed
the
stale
|
I like this idea but would like it to be less maintenance-intensive. |
good idea, could you provide an example on that suggestion? do you have a reference link or would you like to take that up and refine it to be a parameter instead? |
|
Hi @MoeMahhouk, this pull request has been closed automatically due to 30 days of inactivity. If you’d like to continue working on it, feel free to reopen at any time. |
|
Hey folks, @chong-he @jimmygchen ,this PR got auto-closed due to inactivity, but the changes are current and needs review. Could we reopen this for another pass? |
Reopening, could you solve some conflicts in the branch? |
solved the conflicts and merged. Let me know if you want me to do any other change |
| echo "=== Building first Docker image (x86_64) ===" | ||
| docker build -f Dockerfile.reproducible \ | ||
| --build-arg RUST_TARGET="x86_64-unknown-linux-gnu" \ | ||
| --build-arg RUST_IMAGE="rust:1.88-bullseye@sha256:8e3c421122bf4cd3b2a866af41a4dd52d87ad9e315fd2cb5100e87a7187a9816" \ |
There was a problem hiding this comment.
I think we can remove this line and instead just let it read from Dockerfile.reproducible for the RUST_IMAGE. I have tested it on my local repo and it works, see: https://github.com/chong-he/lighthouse/pull/9/files#diff-587298ff141278ce3be7c54a559f9f31472cc5b384e285e2105b3dee319ba31d
This also applies to all other instances that uses this RUST_IMAGE. Taking the idea from Dockerfile:
Line 1 in e3ee7fe
We can also remove the sha256 in the RUST_IMAGE in Dockerfile.reproducible. That will make it much easier to maintain.
We also do not need a separate image on x86_64 or arm64 from my understanding, as docker is able to select the image based on the host: https://docs.docker.com/build/building/multi-platform/#difference-between-single-platform-and-multi-platform-images, so that will make things simpler
| --build-arg RUST_IMAGE="rust:1.88-bullseye@sha256:8e3c421122bf4cd3b2a866af41a4dd52d87ad9e315fd2cb5100e87a7187a9816" \ |
There was a problem hiding this comment.
I am pinning the sha256 for better tracking and reproducibility stability because if any dependency gets updated for that specific image version of rust:1.88, it would also changes its hash history and eventually also impact the reproducibility of the generated lighthouse binary.
If you are ok to proceed with such trade-off, I can remove the sha256 pinning to the rust image.
Regarding removing the whole build-arg, I believe that would be necessary since we are not passing the --platform to the docker build command for the image to identify which one to select.
Nonetheless, if you decide to keep the sha256 for stable reproducibility tracking, then I believe we can't remove this line as it wouldn't be able to detect which sha256 to fetch for which platform, right?
| echo "🔄 Verifying reproducible builds for ${{ matrix.arch }}..." | ||
|
|
||
| # Build first image | ||
| echo "=== Building first verification image ===" |
There was a problem hiding this comment.
I know I raised this before to have a reproducible check before publishing the image. I am sorry, thinking about it now, we already have the reproducible-CI checks, and this seems repetitive now. In the interest of making things simpler, should we remove the check? I am trying to make the changes as minimal as possible, so that it is easier to read and maintain. After all, the checks for reproducibility is running every 2 days so if anything, we can catch it earlier.
I am just considering about the consequence - if without the checks and the docker images are pushed, then it could be that the docker images are not reproducible in the worse outcome?
There was a problem hiding this comment.
we can remove the check but then we would have the scenario that the reproducible-build CI might break but this workflow still builds and publishes a "reproducible" docker build despite being broken.
Do you still want to remove the reproducibility check?
Yes, I recall we discussed this before and decided to add this check here despite repetitiveness to avoid such misleading scenario where shipped reproducible docker builds aren't actually reproducible
Dockerfile.reproducible
Outdated
| FROM gcr.io/distroless/cc-debian12:nonroot-6755e21ccd99ddead6edc8106ba03888cbeed41a | ||
| COPY --from=builder /lighthouse /lighthouse | ||
|
|
||
| EXPOSE 30303 30303/udp 9001 8545 8546 |
There was a problem hiding this comment.
I am not sure if this is necessary? The ports 8545, 8546, 30303 are for the EL
There was a problem hiding this comment.
Sure, which ones do you recommend to replace them with?
There was a problem hiding this comment.
I think we can just delete this line, can't we? When running the image, users will usually specify the port themselves, so this feels redundant
| echo "🔄 Verifying reproducible builds for ${{ matrix.arch }}..." | ||
|
|
||
| # Build first image | ||
| echo "=== Building first verification image ===" |
There was a problem hiding this comment.
we can remove the check but then we would have the scenario that the reproducible-build CI might break but this workflow still builds and publishes a "reproducible" docker build despite being broken.
Do you still want to remove the reproducibility check?
Yes, I recall we discussed this before and decided to add this check here despite repetitiveness to avoid such misleading scenario where shipped reproducible docker builds aren't actually reproducible
| release-summary: | ||
| name: release summary | ||
| runs-on: ubuntu-latest | ||
| needs: [extract-version, verify-and-build, create-manifest] | ||
| if: always() | ||
| steps: | ||
| - name: Report release results | ||
| run: | | ||
| VERSION=${{ needs.extract-version.outputs.VERSION }} | ||
| IMAGE_NAME=${{ env.DOCKER_REPRODUCIBLE_IMAGE_NAME }} | ||
|
|
||
| echo "## 🚀 Reproducible Release Summary for ${VERSION}" | ||
| echo "" | ||
|
|
||
| if [[ "${{ needs.verify-and-build.result }}" == "success" ]]; then | ||
| echo "✅ **Reproducibility Verification & Build**: SUCCESS" | ||
| echo "- All architectures produce identical binaries" | ||
| echo "- Images built and ready for publishing" | ||
| else | ||
| echo "❌ **Reproducibility Verification & Build**: FAILED" | ||
| echo "- Builds are not reproducible OR build failed" | ||
| echo "- Release was blocked" | ||
| fi | ||
|
|
||
| echo "" | ||
| if [[ "${{ github.event.inputs.dry_run }}" == "true" ]]; then | ||
| echo "🧪 **Mode**: DRY RUN" | ||
| echo "- Images were built and verified but NOT pushed" | ||
| echo "- Ready for real release" | ||
| elif [[ "${{ needs.create-manifest.result }}" == "success" ]]; then | ||
| echo "✅ **Publication**: SUCCESS" | ||
| echo "- Images published to Docker Hub" | ||
| echo "- Multi-arch manifests created" | ||
| echo "" | ||
| echo "### 📦 Published Images" | ||
| echo "- \`${IMAGE_NAME}:${VERSION}\`" | ||
| echo "- \`${IMAGE_NAME}:latest\`" | ||
| echo "" | ||
| echo "### 🏗️ Architectures" | ||
| echo "- linux/amd64 (\`${IMAGE_NAME}:${VERSION}-amd64\`)" | ||
| echo "- linux/arm64 (\`${IMAGE_NAME}:${VERSION}-arm64\`)" | ||
| else | ||
| echo "❌ **Publication**: FAILED" | ||
| echo "- Images were verified but failed to publish" | ||
| fi | ||
|
|
||
| echo "" | ||
| if [[ "${{ needs.verify-and-build.result }}" == "success" ]] && [[ "${{ needs.create-manifest.result }}" == "success" ]] && [[ "${{ github.event.inputs.dry_run }}" != "true" ]]; then | ||
| echo "🎉 **Overall**: Secure release completed successfully!" | ||
| echo "" | ||
| echo "### 🔒 Security Guarantees" | ||
| echo "- ✅ Reproducible builds verified" | ||
| echo "- ✅ Identical binaries across architectures" | ||
| echo "- ✅ No build artifacts tampering" | ||
| echo "- ✅ Deterministic build process" | ||
| elif [[ "${{ github.event.inputs.dry_run }}" == "true" ]]; then | ||
| echo "🧪 **Overall**: Dry run completed successfully!" | ||
| echo "- Reproducibility verified ✅" | ||
| echo "- Ready for real release ✅" | ||
| else | ||
| echo "🚨 **Overall**: Release failed or incomplete" | ||
| echo "- Check logs above for details" | ||
| fi |
There was a problem hiding this comment.
@chong-he do you also want me to remove the release summary here too?
There was a problem hiding this comment.
yes I think we can remove the release summary here as well, thanks
| echo "=== Building first Docker image (x86_64) ===" | ||
| docker build -f Dockerfile.reproducible \ | ||
| --build-arg RUST_TARGET="x86_64-unknown-linux-gnu" \ | ||
| --build-arg RUST_IMAGE="rust:1.88-bullseye@sha256:8e3c421122bf4cd3b2a866af41a4dd52d87ad9e315fd2cb5100e87a7187a9816" \ |
There was a problem hiding this comment.
I am pinning the sha256 for better tracking and reproducibility stability because if any dependency gets updated for that specific image version of rust:1.88, it would also changes its hash history and eventually also impact the reproducibility of the generated lighthouse binary.
If you are ok to proceed with such trade-off, I can remove the sha256 pinning to the rust image.
Regarding removing the whole build-arg, I believe that would be necessary since we are not passing the --platform to the docker build command for the image to identify which one to select.
Nonetheless, if you decide to keep the sha256 for stable reproducibility tracking, then I believe we can't remove this line as it wouldn't be able to detect which sha256 to fetch for which platform, right?
Dockerfile.reproducible
Outdated
| FROM gcr.io/distroless/cc-debian12:nonroot-6755e21ccd99ddead6edc8106ba03888cbeed41a | ||
| COPY --from=builder /lighthouse /lighthouse | ||
|
|
||
| EXPOSE 30303 30303/udp 9001 8545 8546 |
There was a problem hiding this comment.
Sure, which ones do you recommend to replace them with?
| # Remove build ID from the binary to ensure reproducibility across builds | ||
| RUST_BUILD_FLAGS += -C link-arg=-Wl,--build-id=none | ||
| # Remove metadata hash from symbol names to ensure reproducible builds | ||
| RUST_BUILD_FLAGS += -C metadata='' |
There was a problem hiding this comment.
why do we need 3 RUST_BUILD_FLAGS here?
(I had this comment in the same time as 2 days ago but probably I forgot to click to publish the comment)
4 participants
Issue Addressed
This pull request introduces workflows and updates to ensure reproducible builds for the Lighthouse project. It adds two GitHub Actions workflows for building and testing reproducible Docker images and binaries, updates the
Makefileto streamline reproducible build configurations, and modifies theDockerfile.reproducibleto align with the new build process. Additionally, it removes thereproducibleprofile fromCargo.toml.Proposed Changes
New GitHub Actions Workflows:
.github/workflows/release-reproducible.yml: Adds a workflow to build and push reproducible multi-architecture Docker images for releases, including support for dry runs..github/workflows/reproducible-build.yml: Adds a workflow to test reproducibility by comparing binaries built twice forx86_64andaarch64architectures, with failure reporting and artifact uploads.Build Configuration Updates:
Makefile: Refactors reproducible build targets, centralizes environment variables for reproducibility, and updates Docker build arguments forx86_64andaarch64architectures.Dockerfile.reproducible: Updates the base Rust image to version 1.86, removes hardcoded reproducibility settings, and delegates build logic to theMakefile.Profile Removal:
Cargo.toml: Removes thereproducibleprofile, simplifying build configurations and relying on external tooling for reproducibility.Additional Info
This is mainly a follow up to this work #6799 where I refine the reproducible build configuration to simplify the CI workflow to generate the reproducible images and pushes them to DockerHub. I also added a cron job workflow (inspired from the Reth repo) that checks every two days or pull requests that touches files that might affect reproducibility to catch potential regressions.
In case, this is too much, let me know and I can create a separate PR for this to be merged later when necessary
close #7486
close #7485