CLOUDP-339241 - re-enable multi arch smoke tests #539

MaciejKaras · 2025-10-20T15:11:25Z

Summary

Key changes include:

Added support for IBM Power and IBM Z architectures with podman builder configuration
Updated repository URLs from quay.io/mongodb/mongodb-kubernetes-tests to 268558157000.dkr.ecr.us-east-1.amazonaws.com/staging/mongodb-kubernetes-tests for release scenarios. This will prevent meko-test image to be available also for our customers
Enhanced build pipeline to support both Docker and Podman builders for different architectures

Proof of Work

Staging job is successful -> https://spruce.mongodb.com/version/69033a1e9701b900075d742f (ignore ibm z which is disabled in updated code)

Release job is successful -> https://spruce.mongodb.com/version/69036de79701b900075df4de (ignore GKE code snippets tasks, they need different fix. create_chart_release_pr will fail due to missing tag)

We are not ready yet to enable smoke tests for IBM Z, because our evergreen machines are unstable -> https://jira.mongodb.org/browse/DEVPROD-23283

Checklist

Have you linked a jira ticket and/or is the ticket in the title?
Have you checked whether your jira ticket required DOCSP changes?
Have you added changelog file?
- use skip-changelog label if not needed
- refer to Changelog files and Release Notes section in CONTRIBUTING.md for more details

github-actions · 2025-10-20T15:12:37Z

⚠️ (this preview might not be accurate if the PR is not rebased on current master branch)

MCK 1.6.0 Release Notes

New Features

MongoDBCommunity: Added support to configure custom cluster domain via newly introduced spec.clusterDomain resource field. If spec.clusterDomain is not set, environment variable CLUSTER_DOMAIN is used as cluster domain. If the environment variable CLUSTER_DOMAIN is also not set, operator falls back to cluster.local as default cluster domain.
Helm Chart: Introduced two new helm fields operator.podSecurityContext and operator.securityContext that can be used to configure securityContext for Operator deployment through Helm Chart.

Bug Fixes

Fixed parsing of the customEnvVars Helm value when values contain = characters.
ReplicaSet: Blocked disabling TLS and changing member count simultaneously. These operations must now be applied separately to prevent configuration inconsistencies.

Other Changes

kubectl-mongodb plugin: cosign, the signing tool that is used to sign kubectl-mongodb plugin binaries, has been updated to version 3.0.2. With this change, released binaries will be bundled with .bundle files containing both signature and certificate information. For more information on how to verify signatures using new cosign version please refer to -> https://github.com/sigstore/cosign/blob/v3.0.2/doc/cosign_verify-blob.md

Copilot

Pull Request Overview

This PR re-enables multi-architecture (IBM Z and IBM Power) smoke tests for release builds by migrating the meko-test image from the public Quay.io registry to a staging ECR repository. This prevents test images from being publicly available to customers during release processes.

Key changes include:

Added support for IBM Power and IBM Z architectures with podman builder configuration
Updated repository URLs from quay.io/mongodb/mongodb-kubernetes-tests to 268558157000.dkr.ecr.us-east-1.amazonaws.com/staging/mongodb-kubernetes-tests for release scenarios
Enhanced build pipeline to support both Docker and Podman builders for different architectures

Reviewed Changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`scripts/release/tests/build_info_test.py`	Updates test configurations to include IBM Power and Z architectures with staging ECR repositories
`scripts/release/pipeline.py`	Adds new image constants and builder support, updates argument parsing for skip-if-exists flag
`scripts/release/build/build_info.py`	Adds builder constants and integrates builder field into ImageInfo configuration
`scripts/release/atomic_pipeline.py`	Implements podman build support alongside existing Docker functionality
`build_info.json`	Defines multi-arch image configurations with staging repositories for release scenarios
`.evergreen.yml`	Updates build variants and tasks to support separate IBM Power and Z builds
`.evergreen-release.yml`	Creates dedicated release variants for multi-arch smoke testing

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

scripts/release/atomic_pipeline.py

scripts/dev/recreate_python_venv.sh

lucian-tosa · 2025-11-04T13:08:49Z

scripts/release/atomic_pipeline.py

+    if build_configuration.builder == BUILDER_PODMAN:
+        logger.info(
+            f"Building image with podman, tags {tags} for platforms={build_configuration.platforms}, dockerfile args: {build_args}"
+        )
+        try:
+            build_command = [
+                "sudo",
+                "podman",
+                "buildx",
+                "build",
+                "--progress",
+                "plain",
+                build_path,
+                "-f",
+                build_configuration.dockerfile_path,
+            ]
+            for tag in tags:
+                build_command.extend(["-t", tag])
+            for key, value in build_args.items():
+                build_command.extend(["--build-arg", f"{key}={value}"])
+
+            result = subprocess.run(build_command, capture_output=True, text=True, check=True)
+            logger.debug(result.stdout)
+
+            for tag in tags:
+                push_command = ["sudo", "podman", "push", "--authfile=/root/.config/containers/auth.json", tag]
+                result = subprocess.run(push_command, capture_output=True, text=True, check=True)
+                logger.debug(result.stdout)


This logic seems better suited for execute_docker_build

Probably not execute_docker_build because it's using docker ;) but I agree this can be encapsulated in separate method with similar signature as execute_docker_build. Or you meant something else?

lucian-tosa · 2025-11-04T13:11:43Z

scripts/release/atomic_pipeline.py

+            build_command = [
+                "sudo",
+                "podman",
+                "buildx",
+                "build",
+                "--progress",
+                "plain",
+                build_path,
+                "-f",
+                build_configuration.dockerfile_path,


Looks like the platform flag is not used (except for the tag). I know it is implied by the machine it runs on, but for clarity we should set it here

I don't see the platform flag being used before:

sudo podman buildx build --progress plain . -f Dockerfile -t "${REGISTRY}/mongodb-kubernetes-tests:${OPERATOR_VERSION}-$(arch)" --build-arg PYTHON_VERSION="${PYTHON_VERSION}"

lucian-tosa · 2025-11-04T13:14:32Z

.evergreen-release.yml

+      - release-rhel9-power-small # This is required for CISA attestation https://jira.mongodb.org/browse/DEVPROD-17780
+      - release-rhel9-power-large # This is required for CISA attestation https://jira.mongodb.org/browse/DEVPROD-17780


it's a private test image, why?

well it's not a big deal, but we are using this test image to verify that our released images are passing smoke tests. In theory somebody from outside could interfere with test image build process and acquire stored credentials for quay.io for instance

lucian-tosa · 2025-11-04T13:18:42Z

scripts/dev/contexts/e2e_smoke_ibm_power_release

+export CUSTOM_OM_VERSION
+
+# During release the meko-tests image is pushed to a staging registry. We don't want to push it to production registry
+export MEKO_TESTS_REGISTRY="268558157000.dkr.ecr.us-east-1.amazonaws.com/staging"


Shouldn't this logic be in evg-private-context? We will never use quay for test image, so I think this can be simplified

yeah, this can be also in evg-private-context. In the future evg-private-context should read repository settings from build_info.json instead of having them hardcoded. If you have nothing against this I would rather target this in separate PR and use build_info.json properly.

anandsyncs · 2025-11-04T21:11:32Z

scripts/release/tests/build_info_test.py

+                architecture_suffix=True,
+            ),
+            "meko-tests-ibm-z": ImageInfo(
+                repositories=["268558157000.dkr.ecr.us-east-1.amazonaws.com/dev/mongodb-kubernetes-tests"],


nit: It would be better to use constants for the repetitive dev and staging repos.

anandsyncs

LGTM, there is one nit and another issue the Lucian already mentioned:
https://github.com/mongodb/mongodb-kubernetes/pull/539/files#r2490868628

MaciejKaras added the skip-changelog Use this label in Pull Request to not require new changelog entry file label Oct 20, 2025

MaciejKaras force-pushed the maciejk/ar-image-release-smoke-tests branch from 411ad0d to 0937335 Compare October 29, 2025 15:40

MaciejKaras requested a review from Copilot October 30, 2025 10:19

Copilot AI reviewed Oct 30, 2025

View reviewed changes

scripts/release/atomic_pipeline.py Show resolved Hide resolved

scripts/release/atomic_pipeline.py Outdated Show resolved Hide resolved

scripts/dev/recreate_python_venv.sh Show resolved Hide resolved

MaciejKaras force-pushed the maciejk/ar-image-release-smoke-tests branch from 04bdb0a to 0dbe506 Compare October 30, 2025 10:20

CLOUDP-339241 - re-enable multi arch smoke tests

19d5534

MaciejKaras force-pushed the maciejk/ar-image-release-smoke-tests branch from 0dbe506 to 19d5534 Compare October 30, 2025 10:28

MaciejKaras added 3 commits October 30, 2025 12:03

Disable ibm z completely

8252467

Fix release tasks

44307ae

Fix release tasks

97333b2

MaciejKaras marked this pull request as ready for review October 31, 2025 08:31

MaciejKaras requested a review from a team as a code owner October 31, 2025 08:31

MaciejKaras requested review from anandsyncs, lucian-tosa, nammn and viveksinghggits and removed request for nammn October 31, 2025 08:31

lucian-tosa requested changes Nov 4, 2025

View reviewed changes

anandsyncs reviewed Nov 4, 2025

View reviewed changes

anandsyncs approved these changes Nov 4, 2025

View reviewed changes

		- release-rhel9-power-small # This is required for CISA attestation https://jira.mongodb.org/browse/DEVPROD-17780
		- release-rhel9-power-large # This is required for CISA attestation https://jira.mongodb.org/browse/DEVPROD-17780

CLOUDP-339241 - re-enable multi arch smoke tests #539

Are you sure you want to change the base?

CLOUDP-339241 - re-enable multi arch smoke tests #539

Conversation

MaciejKaras commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Proof of Work

Checklist

Uh oh!

github-actions bot commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

MCK 1.6.0 Release Notes

New Features

Bug Fixes

Other Changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anandsyncs left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

MaciejKaras commented Oct 20, 2025 •

edited

Loading

github-actions bot commented Oct 20, 2025 •

edited

Loading