Skip to content

ci: add engine-agnostic smoke-service cleanup task - #544

Draft
wz-gsa wants to merge 1 commit into
mainfrom
chore/ci-smoke-service-cleanup
Draft

wz-gsa wants to merge 1 commit into
mainfrom
chore/ci-smoke-service-cleanup

Conversation

@wz-gsa

@wz-gsa wz-gsa commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Split out of #537 (engine-agnostic CI hygiene, not Oracle-specific).

Adds a reusable Concourse task (ci/cleanup-smoke-service.yml + .sh) that tears down a leaked smoke-test service instance. It's engine-agnostic so any RDS (or other) smoke job can wire it as an ensure: step to avoid leaking a live instance when a smoke test fails partway.

Standalone task only here; wiring it into a specific job's ensure: block is done by the PR that owns that job (e.g. the Oracle smoke task in #537).

@wz-gsa
wz-gsa requested a review from a team as a code owner July 21, 2026 18:42
@wz-gsa
wz-gsa force-pushed the chore/ci-smoke-service-cleanup branch from 01d8ecd to d461087 Compare July 21, 2026 20:45
@wz-gsa

wz-gsa commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

Ran a hostile self-review and fixed the findings (pushed):

HIGH — the reaper missed most leaks it exists for. The old regex smoke-tests.*<plan> (plural) didn't match the singular smoke-test-<plan>-* instances that the task/ES/unbound runners create — exactly the billed instances it's meant to reap — while staying green. Rewrote matching to be anchored + token-aware so it reaps every shape the runners produce (both smoke-test-<plan>-* and *-smoke-tests-...-<plan>[-<id>]). Verified all 9 runner name shapes match for several plans (incl. the Oracle plan).

MEDIUM — over-broad match / collateral deletion. The old unanchored substring could force-delete a concurrent related-plan run (e.g. redis-dev matching redis-dev-large). The new pattern is anchored with a token boundary, and SERVICE_PLAN is escaped to a literal before going into the ERE. Verified redis-dev no longer matches ...-redis-dev-large.

MEDIUM — auth failure masked into a permanent green no-op. login || true swallowed a failed cf login, so the reaper would silently do nothing forever. Now a login failure emits a loud, greppable WARN ... NOT reaped and exits 0 (still best-effort, still never fails the build).

Also clarified the delete-service scope (removes the instance's bindings/keys, not the separate test app) and documented that the ensure: wiring + CF_ORGANIZATION/CF_SPACE/SERVICE_PLAN params (which must target the smoke-test space) are added by the PR that owns the specific smoke job. shellcheck + shfmt clean.

Add a reusable Concourse task (cleanup-smoke-service.yml + .sh) that tears down
leaked smoke-test SERVICE INSTANCES for a given SERVICE_PLAN, engine-agnostic so
any RDS/Elasticache/Elasticsearch smoke job can wire it as an `ensure:` step to
avoid leaking a live billed instance on failure.

Standalone task only; the `ensure:` wiring (and its CF_ORGANIZATION/CF_SPACE +
SERVICE_PLAN params, which must target the smoke-test space) is added by the PR
that owns the specific smoke job.

Matching is anchored and token-aware so it reaps every instance the smoke runners
create for THIS plan — both singular `smoke-test-<plan>-*` (task/es runners) and
plural `*-smoke-tests-...-<plan>[-<id>]` (rds/redis runners) — without deleting a
DIFFERENT plan's instance whose name merely contains the plan as a substring
(e.g. plan `redis-dev` will not reap `...-redis-dev-large`). SERVICE_PLAN is
escaped to a literal before going into the ERE. A cf login failure is surfaced as
a loud WARNING and the reaper exits 0 (best-effort, never fails the build) rather
than silently doing nothing.
@wz-gsa
wz-gsa force-pushed the chore/ci-smoke-service-cleanup branch from d461087 to fcca28b Compare July 21, 2026 20:59
@pburkholder
pburkholder marked this pull request as draft July 22, 2026 14:26
@wz-gsa

wz-gsa commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

📋 Merge order: 3️⃣ of 4 (adds ci/ task files but does NOT wire them into ci/pipeline.yml → pipeline-inert until a later PR references them).
Coordinated set; full order + status board: #550

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant