ci: isolate integration tests in per-run CF spaces#1215
Open
kbarnold wants to merge 5 commits intoci/docker-image-buildsfrom
Open
ci: isolate integration tests in per-run CF spaces#1215kbarnold wants to merge 5 commits intoci/docker-image-buildsfrom
kbarnold wants to merge 5 commits intoci/docker-image-buildsfrom
Conversation
Concurrent pipeline runs (PR builds, master merges, release tags) all deployed to the same hardcoded CF space, causing race conditions where one run's cleanup tore down another's deployment mid-flight. The test job now creates a unique mbt-ci-<build-num> space, passes it to the test process, and deletes it with when: always to prevent orphaned spaces.
Cancelled pipeline runs can leave mbt-ci-* spaces behind because the always-run cleanup step never executes. This adds a cleanup-ci-cf-spaces job that deletes any leftover mbt-ci-* spaces and a nightly-cleanup workflow triggered via cron at 02:00 UTC.
The cleanup job was incorrectly placed inside a workflow definition block instead of the top-level jobs section, making it invisible to the nightly-cleanup workflow that references it.
The NODE_APP_ROUTE env var contained a hardcoded URL with the old fixed space name. Since CF routes now include the dynamic space name, the test queries the actual route from cf app output after deployment instead of relying on the environment variable.
- Guard the cleanup step against empty TEST_SPACE: if space creation failed (e.g. quota exceeded) TEST_SPACE is never written to BASH_ENV, causing `cf delete-space "" -f` to fail or match incorrectly; skip gracefully instead - Nightly cleanup now skips spaces younger than 2 hours to avoid deleting the space of a currently running integration-test pipeline; uses portable date arithmetic (GNU and BSD date both supported) - Update test job Docker image from cimg/go:1.21-node to cimg/go:1.24-node to match the Go 1.24 version used in the project's Dockerfile - Fix malformed CF CLI download URL: replace `&&version=` with `&version=`
kbarnold
force-pushed
the
ci/docker-image-builds
branch
from
864403e to
c213374
Compare
kbarnold
force-pushed
the
ci/cf-test-isolation
branch
from
2ffe622 to
092e70a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
mbt-ci-<CIRCLE_BUILD_NUM>CF space per pipeline run, eliminating race conditions when concurrent pipelines share a spacewhen: always, so it is deleted even on test failureTEST_SPACE(e.g. if space creation failed), avoiding acf delete-space "" -fcallcleanup-ci-cf-spacesjob that deletes orphanedmbt-ci-*spaces, skipping any space younger than 2 hours to avoid racing with a currently running pipelinecf app nodeoutput instead of a hardcodedNODE_APP_ROUTEenv var (the route now includes the dynamic space name)testjob Docker image fromcimg/go:1.21-nodetocimg/go:1.24-nodeto match the Go version used in the project Dockerfile&&version=→&version=Why
Every CI run previously deployed to the same CF space. When two pipelines ran concurrently, their
cf undeploy/cf deletecalls raced against each other, causing random test failures. Per-run spaces provide full isolation.Test plan
TEST_SPACE)