Skip to content

Commit a74c1af

Browse files
nogueiraandersonnogueiraanderson
authored
PG-2353: Auto-prune superseded AMIs after each promote (#4213)
- After each prod promote the workflow prunes that (os, os_major, arch) combo with scripts/prune-superseded.sh, using the same pin floor as just prune-superseded: the combo's pin in vars/moleculeEnvPPG.groovy on master and every newer AMI survive, older ones are deregistered with their snapshots - The script refuses to prune blind: a describe failure, a pins file whose 12 entries do not each parse exactly once, a pin missing from the candidates, a non-production role, or a just-promoted AMI still not the newest visible match after a bounded wait all abort - Images pinned by any combo or named by an open refresh PR are never pruned, an apply run needs the pins file named explicitly and exits non-zero on any failed deregister or snapshot error (skipped and missing snapshots are benign) - The workflow prune step runs for prod bakes only, fetches master's pins and every open refresh PR's pins at prune time, stops when the PR listing fails, and fails the leg instead of continue-on-error so the notify job pages - _deregister uses --delete-associated-snapshots and exits non-zero on a snapshot error, _prune-by-filter and prune-stale count failed deregisters instead of stopping at the first, prune-stale describes once and skips demoted *superseded* AMIs - The Session Manager plugin is pinned to a versioned URL with a SHA256 check, and a smoke packer init failure keeps the candidate - The smoke template validates os_major and arch
1 parent 76e0a32 commit a74c1af

6 files changed

Lines changed: 507 additions & 34 deletions

File tree

‎.github/workflows/ppg-ami-factory.yml‎

Lines changed: 59 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -103,10 +103,18 @@ jobs:
103103
aws-region: ${{ env.AWS_REGION }}
104104

105105
- name: Install Session Manager plugin
106+
env:
107+
# Pinned to an immutable versioned URL + verified SHA256 (supply-chain): never
108+
# `latest`. The plugin is installed in a job holding AWS OIDC creds, so the bytes
109+
# are integrity-checked. Bump deliberately and re-record the digest of the new
110+
# version's .deb (curl the versioned URL, sha256sum it).
111+
SMP_VERSION: 1.2.835.0
112+
SMP_SHA256: 7c6dcad12518571cc7959a713e6a8ae1bdf6ed66fd9bee37dc189e39ca58ae03
106113
run: |
107114
set -euo pipefail
108115
if ! command -v session-manager-plugin >/dev/null; then
109-
curl -fsSL "https://s3.amazonaws.com/session-manager-downloads/plugin/latest/ubuntu_64bit/session-manager-plugin.deb" -o /tmp/smp.deb
116+
curl -fsSL "https://s3.amazonaws.com/session-manager-downloads/plugin/${SMP_VERSION}/ubuntu_64bit/session-manager-plugin.deb" -o /tmp/smp.deb
117+
echo "${SMP_SHA256} /tmp/smp.deb" | sha256sum -c -
110118
sudo dpkg -i /tmp/smp.deb
111119
fi
112120
session-manager-plugin --version
@@ -161,8 +169,11 @@ jobs:
161169
run: |
162170
set -euo pipefail
163171
echo "smoke candidate: $AMI"
172+
# `packer init` is CI infra (plugin download); a transient init failure must NOT
173+
# deregister a good candidate. Only a real boot/install (build) failure does.
164174
# Deregister ONLY on a real boot/install failure (never on a later promote-tag failure).
165-
( cd smoke && packer init . && packer build -color=false \
175+
( cd smoke && packer init . ) || { echo "smoke packer init failed (CI infra, not a candidate defect); keeping $AMI"; echo "- smoke: init failed (kept $AMI)" >> "$GITHUB_STEP_SUMMARY"; exit 1; }
176+
( cd smoke && packer build -color=false \
166177
-var "candidate_ami=$AMI" -var "os=${OS}" -var "os_major=${OS_MAJOR}" \
167178
-var "arch=${ARCH}" -var "region=${AWS_REGION}" . ) \
168179
|| { echo "smoke (boot+install) failed; deregistering $AMI + its snapshots"; aws ec2 deregister-image --delete-associated-snapshots --region "$AWS_REGION" --image-id "$AMI"; echo "- smoke: FAIL (deregistered $AMI)" >> "$GITHUB_STEP_SUMMARY"; exit 1; }
@@ -183,6 +194,52 @@ jobs:
183194
done
184195
echo "- promote: $pr" >> "$GITHUB_STEP_SUMMARY"
185196
197+
- name: Prune superseded (older than the master pin)
198+
# Prod only: the test role has no consumer pins to floor on, and its
199+
# images are cleaned by `just prune-test`.
200+
if: env.FACTORY_ENV == 'prod'
201+
working-directory: ppg/packer
202+
# Promote already ran, so a prune failure cannot un-ship the AMI: it
203+
# fails only this leg (fail-fast is off) and pages via the notify job.
204+
# No continue-on-error: a step failing under it keeps the job green
205+
# (outcome=failure, conclusion=success), which makes the notify job's
206+
# failure() unreachable. The script appends counts to
207+
# $GITHUB_STEP_SUMMARY and exits non-zero on any failed deregister,
208+
# failed snapshot cleanup, or image older than the pin left behind.
209+
env:
210+
OS: ${{ matrix.os || 'oraclelinux' }}
211+
OS_MAJOR: ${{ matrix.os_major }}
212+
ARCH: ${{ matrix.arch }}
213+
AMI: ${{ steps.build.outputs.ami }}
214+
GH_TOKEN: ${{ github.token }}
215+
run: |
216+
set -euo pipefail
217+
# Deregister this combo's promoted bases older than master's pin. The pin and
218+
# every newer image survive (same rule as `just prune-superseded`).
219+
# deprecate_at only marks, it never deletes, so without this the inventory grows.
220+
: "${AMI:?build output ami is empty, refusing to prune unguarded}"
221+
# PROMOTED_AMI guards the prune against EC2 eventual consistency: the script
222+
# refuses to act unless the just-promoted AMI is the newest visible match.
223+
# OS_NAME scopes the prune to this matrix leg's os so combos sharing an
224+
# os_major + arch (oraclelinux 9 vs rocky 9) never prune each other.
225+
# The pins come from master at prune time, not from this run's checkout,
226+
# so a re-run of an old workflow run cannot protect stale pins.
227+
git fetch --quiet --depth 1 origin master
228+
git show FETCH_HEAD:vars/moleculeEnvPPG.groovy > "${RUNNER_TEMP}/pins.groovy"
229+
# Images named by open same-repo refresh PRs are protected too: merging
230+
# such a PR must never point pg.cd at a deregistered image.
231+
# The listing is assigned first so a gh failure stops the step (set -e)
232+
# instead of pruning with no PR protection.
233+
open_heads=$(gh api --paginate "repos/${GITHUB_REPOSITORY}/pulls?state=open&base=master&per_page=100" \
234+
--jq '.[] | select(.head.repo.owner.login == "'"${GITHUB_REPOSITORY_OWNER}"'") | select(.head.ref | test("^ppg-ami-refresh(-[0-9]{8})?$")) | .head.ref')
235+
protect_files=""
236+
for head in ${open_heads}; do
237+
git fetch --quiet --depth 1 origin "$head"
238+
git show FETCH_HEAD:vars/moleculeEnvPPG.groovy > "${RUNNER_TEMP}/pins-${head}.groovy"
239+
protect_files="${protect_files:+${protect_files}:}${RUNNER_TEMP}/pins-${head}.groovy"
240+
done
241+
PINS_FILE="${RUNNER_TEMP}/pins.groovy" PROTECT_FILES="$protect_files" PROMOTED_AMI="$AMI" OS_NAME="$OS" bash scripts/prune-superseded.sh ppg-package-test "$OS_MAJOR" "$ARCH" 1 "$AWS_REGION"
242+
186243
update-molecule-env:
187244
name: Update moleculeEnvPPG.groovy (OL/Rocky AMI IDs)
188245
needs: bake

‎ppg/packer/README.md‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -85,13 +85,25 @@ is the newest `role=ppg-package-test` AMI by `CreationDate` (no SSM parameter).
8585

8686
## Housekeeping (all via `just`, fail-safe)
8787

88-
Cleanup is recipes too. Every prune recipe **lists by default** and deregisters
89-
only on an explicit `1`; the guard **never deletes a promoted prod base** and
90-
fail-closes on an AMI it cannot positively classify.
88+
Superseded prod bases are pruned **automatically**: after each successful promote the
89+
workflow runs `scripts/prune-superseded.sh` for that combo. The floor is the combo's pin
90+
in `vars/moleculeEnvPPG.groovy` on master. The pin and every newer AMI survive, and so
91+
does any image pinned by another combo or named by an open refresh PR. Only images older
92+
than the pin go. `deprecate_at` only marks an AMI deprecated, it never deletes, so without
93+
this the inventory grows with every refresh. `just prune-superseded` applies the same
94+
rule to all combos for ad-hoc / backfill cleanup.
95+
96+
Every prune recipe **lists by default** and deregisters only on an explicit `1`. The guard
97+
**never deletes a pinned base or anything newer** and fail-closes on an AMI it cannot classify.
98+
Demoted rollback AMIs (`role=*-superseded*`) are never touched by any recipe. The
99+
post-promote prune additionally refuses to act while the just-promoted AMI is not yet
100+
visible as the newest of its combo, and reports every skipped deregister, failed
101+
snapshot cleanup, or image older than the pin left behind as a workflow warning +
102+
step-summary line.
91103

92104
```bash
93105
just list # current factory AMIs (prod|test)
94-
just prune-superseded # older prod dups (keeps newest per combo); add 1 to delete
106+
just prune-superseded # prod bases older than the master pin; add 1 to delete
95107
just prune-test # isolated env=test AMIs; add 1 to delete
96108
just prune-stale # raw/candidate intermediates + orphans; add 1 to delete
97109
just prune-all # prune-test + prune-stale

‎ppg/packer/justfile‎

Lines changed: 78 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -80,18 +80,36 @@ _promote ami role:
8080
8181
# deregister an AMI + delete its backing snapshots. Idempotent (an already-gone AMI
8282
# is a no-op) but NOT error-masking: a genuine API failure aborts rather than silently
83-
# leaking a snapshot. describe-images --image-ids errors InvalidAMIID.NotFound when gone.
83+
# leaking a snapshot, and a non-success snapshot result exits non-zero after being
84+
# reported, so caller loops can count it instead of ending on a silent success.
8485
_deregister ami:
8586
#!/usr/bin/env bash
8687
set -euo pipefail
87-
err=$(mktemp)
88-
snaps=$(aws ec2 describe-images --profile {{profile}} --region {{region}} --image-ids "{{ami}}" \
89-
--query 'Images[0].BlockDeviceMappings[].Ebs.SnapshotId' --output text 2>"$err") || {
90-
if grep -qiE "InvalidAMIID.NotFound|does not exist" "$err"; then echo " {{ami}} already gone"; rm -f "$err"; exit 0; fi
91-
echo " ERROR describing {{ami}}:" >&2; cat "$err" >&2; rm -f "$err"; exit 1; }
92-
rm -f "$err"
93-
aws ec2 deregister-image --profile {{profile}} --region {{region}} --image-id "{{ami}}"
94-
for s in $snaps; do [ "$s" = None ] && continue; aws ec2 delete-snapshot --profile {{profile}} --region {{region}} --snapshot-id "$s"; done
88+
export AWS_RETRY_MODE="${AWS_RETRY_MODE:-standard}" AWS_MAX_ATTEMPTS="${AWS_MAX_ATTEMPTS:-8}"
89+
error_file=$(mktemp)
90+
snapshot_failures=$(aws ec2 deregister-image --profile {{profile}} --region {{region}} \
91+
--image-id "{{ami}}" --delete-associated-snapshots --output text \
92+
--query "DeleteSnapshotResults[?ReturnCode != 'success'].[SnapshotId, ReturnCode]" 2>"$error_file") || {
93+
if grep -qiE "InvalidAMIID.NotFound|InvalidAMIID.Unavailable|does not exist" "$error_file"; then
94+
echo " {{ami}} already gone"
95+
rm -f "$error_file"
96+
exit 0
97+
fi
98+
99+
echo " ERROR deregistering {{ami}}:" >&2
100+
cat "$error_file" >&2
101+
rm -f "$error_file"
102+
exit 1
103+
}
104+
rm -f "$error_file"
105+
while IFS= read -r snapshot_result; do
106+
[[ -z "$snapshot_result" ]] && continue
107+
echo " WARN snapshot cleanup for {{ami}}: $snapshot_result" >&2
108+
done <<< "$snapshot_failures"
109+
110+
if [[ -n "$snapshot_failures" ]]; then
111+
exit 1
112+
fi
95113
96114
# deregister every self-owned AMI matching a tag filter (+ snapshots), with a HARD guard
97115
# that NEVER touches a promoted prod base (role={{role_prod}}) or an AMI it cannot classify.
@@ -102,6 +120,7 @@ _prune-by-filter apply +filters:
102120
ids=$(aws ec2 describe-images --profile {{profile}} --region {{region}} --owners self \
103121
{{filters}} --query 'Images[].ImageId' --output text)
104122
[ -z "$ids" ] && { echo " (nothing matches)"; exit 0; }
123+
deregister_failures=0
105124
for ami in $ids; do
106125
role=$(aws ec2 describe-images --profile {{profile}} --region {{region}} --image-ids "$ami" \
107126
--query "Images[0].Tags[?Key=='role']|[0].Value" --output text)
@@ -110,9 +129,20 @@ _prune-by-filter apply +filters:
110129
if [ "$role" = "{{role_prod}}" ] || [ -z "$role" ] || [ "$role" = None ]; then echo " SKIP $ami (protected: role=${role:-<none>})"; continue; fi
111130
# fail-safe: deregister ONLY on an exact apply=1; any other value (incl a malformed
112131
# `apply=1` arg, which just passes as the literal "apply=1") lists without deleting.
113-
if [ "{{apply}}" = "1" ]; then echo " deregister $ami (role=$role)"; just _deregister "$ami"; else echo " would deregister $ami (role=$role)"; fi
132+
# || keeps the sweep going: one failed deregister must not skip the rest.
133+
if [ "{{apply}}" = "1" ]; then
134+
echo " deregister $ami (role=$role)"
135+
just _deregister "$ami" || deregister_failures=$((deregister_failures + 1))
136+
else
137+
echo " would deregister $ami (role=$role)"
138+
fi
114139
done
115140
141+
if (( deregister_failures > 0 )); then
142+
echo " ${deregister_failures} deregister(s) failed" >&2
143+
exit 1
144+
fi
145+
116146
# ===========================================================================
117147
# packer build / validate (OL + Rocky refresh, Rocky lineage roots via seed-rocky)
118148
# ===========================================================================
@@ -270,22 +300,46 @@ prune-stale apply="0":
270300
#!/usr/bin/env bash
271301
set -euo pipefail
272302
echo "prune-stale (apply={{apply}}; lists unless apply=1):"
273-
ids=$(aws ec2 describe-images --profile {{profile}} --region {{region}} --owners self \
303+
# one describe returns id + role + factory_env together (no per-AMI
304+
# re-describes); the command-sub still propagates a describe failure under
305+
# set -e. The || 'None' sentinels map an absent tag AND an empty tag value
306+
# to the same token, so a field can never be empty and the tab-split can
307+
# never shift a value into the wrong guard.
308+
candidates=$(aws ec2 describe-images --profile {{profile}} --region {{region}} --owners self \
274309
--filters Name=tag:iit-billing-tag,Values={{billing_tag}} Name=tag:os,Values={{os_csv}} \
275-
--query 'Images[].ImageId' --output text)
276-
[ -z "$ids" ] && { echo " (no factory AMIs)"; exit 0; }
277-
for ami in $ids; do
278-
# two scalar command-subs (not a process-sub `read`): a describe failure propagates
279-
# under set -e instead of being silently swallowed, and no IFS field-shift on tag values.
280-
role=$(aws ec2 describe-images --profile {{profile}} --region {{region}} --image-ids "$ami" \
281-
--query "Images[0].Tags[?Key=='role']|[0].Value" --output text)
282-
env=$(aws ec2 describe-images --profile {{profile}} --region {{region}} --image-ids "$ami" \
283-
--query "Images[0].Tags[?Key=='factory_env']|[0].Value" --output text)
310+
--query "Images[].[ImageId, (Tags[?Key=='role']|[0].Value) || 'None', (Tags[?Key=='factory_env']|[0].Value) || 'None']" --output text)
311+
[[ -z "$candidates" ]] && { echo " (no factory AMIs)"; exit 0; }
312+
deregister_failures=0
313+
while IFS=$'\t' read -r ami role env; do
314+
[[ -z "$ami" ]] && continue
284315
# fail-closed: never delete the prod base, nor anything we cannot positively classify.
285-
if [ "$role" = "{{role_prod}}" ] || [ -z "$role" ] || [ "$role" = None ]; then echo " SKIP $ami (protected: role=${role:-<none>})"; continue; fi
286-
if [ "$env" = "test" ]; then echo " SKIP $ami (test AMI -> use prune-test)"; continue; fi
287-
if [ "{{apply}}" = "1" ]; then echo " deregister $ami (role=$role env=$env)"; just _deregister "$ami"; else echo " would deregister $ami (role=$role env=$env)"; fi
288-
done
316+
if [ "$role" = "{{role_prod}}" ] || [ -z "$role" ] || [ "$role" = None ]; then
317+
echo " SKIP $ami (protected: role=${role:-<none>})"
318+
continue
319+
fi
320+
# demoted rollback AMIs (e.g. role=ppg-ol10-oversized-superseded) are deliberate keeps,
321+
# not stale intermediates; only an explicit human decision removes them.
322+
case "$role" in
323+
*superseded*)
324+
echo " SKIP $ami (protected demotion: role=$role)"
325+
continue
326+
;;
327+
esac
328+
if [ "$env" = "test" ]; then
329+
echo " SKIP $ami (test AMI -> use prune-test)"
330+
continue
331+
fi
332+
if [ "{{apply}}" = "1" ]; then
333+
echo " deregister $ami (role=$role env=$env)"
334+
just _deregister "$ami" || deregister_failures=$((deregister_failures + 1))
335+
else
336+
echo " would deregister $ami (role=$role env=$env)"
337+
fi
338+
done <<< "$candidates"
339+
if (( deregister_failures > 0 )); then
340+
echo " ${deregister_failures} deregister(s) failed" >&2
341+
exit 1
342+
fi
289343
290344
# list ALL factory housekeeping AMIs (test + stale intermediates); never prod. Pass `1` to delete.
291345
prune-all apply="0":

‎ppg/packer/refresh.pkr.hcl‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -195,8 +195,10 @@ source "amazon-ebs" "el" {
195195
skip_profile_validation = true
196196
user_data = local.ssm_bootstrap
197197

198-
# Native retention: the baked AMI auto-deprecates ~5 weeks out, so superseded
199-
# weekly images age out without a custom deregister-old sweep.
198+
# Mark the AMI deprecated ~5 weeks out (deprioritizes it in default describe-images
199+
# results). Deprecation only MARKS, it never deletes the AMI or its snapshot. Superseded
200+
# bases older than the master pin are removed by the post-promote prune
201+
# (scripts/prune-superseded.sh) in the workflow, or by `just prune-superseded`.
200202
deprecate_at = timeadd(timestamp(), "840h")
201203

202204
# Refresh: latest self-owned lineage base (see local.lineage_filters).

0 commit comments

Comments
 (0)