fix: prevent fresh+reconsolidation item name collision in CEFS consolidation#2059
Merged
mattgodbolt merged 2 commits intomainfrom Apr 10, 2026
Merged
fix: prevent fresh+reconsolidation item name collision in CEFS consolidation#2059mattgodbolt merged 2 commits intomainfrom
mattgodbolt merged 2 commits intomainfrom
Conversation
…idation
## Problem
CEFS consolidation failed with:
OSError: [Errno 39] Directory not empty:
.tmp_extract_.../compilers_swift_static-sdk_sdk-0-1-0_6.2.4
-> compilers_swift_static-sdk_sdk-0-1-0_6.2.4
Root cause: a fresh item ('compilers/swift/static-sdk/sdk-0-1-0 6.2.4') and a
reconsolidation item with the same name (from an existing consolidated image)
were both included in the same consolidation group. Both produce the same
sanitized subdir_name ('compilers_swift_static-sdk_sdk-0-1-0_6.2.4'). When
parallel workers extracted them, the second worker found the directory already
written by the first.
## Fix
Two-layer defence:
1. **cli/cefs.py** (prevent): When assembling candidates, filter out
reconsolidation items whose name already appears as a fresh item. The fresh
item supersedes the reconsolidation item since it represents the current
installed state.
2. **consolidation.py** (detect): Add a duplicate subdir_name check in
`prepare_consolidation_items` that raises `ValueError` if two items in
the same group would produce the same extraction directory. This is a safety
net for any future code path that might assemble a bad group.
## Test
New test `test_prepare_consolidation_items_rejects_duplicate_names` reproduces
the exact scenario from production: a fresh Swift static SDK item and a
reconsolidation item with the same name in the same group. The test first
confirms the bug (no exception raised before fix), then confirms the fix
(ValueError raised after).
🤖 Generated by LLM (Claude, via OpenClaw)
Make the test assertion explicit: expect ValueError with a specific message, so a reader can see exactly what wrong behaviour is being guarded against. 🤖 Generated by LLM (Claude, via OpenClaw)
Contributor
There was a problem hiding this comment.
Copilot wasn't able to review any files in this pull request.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
mattgodbolt-molty
added a commit
that referenced
this pull request
Apr 11, 2026
… images ## Problem CEFS consolidation failed again (2026-04-11, run #24277747674) with: ValueError: Duplicate subdir name 'compilers_swift_post-6-2_6.2.4' in consolidation group: 'compilers/swift/post-6-2 6.2.4' (from_reconsolidation=True) conflicts with 'compilers/swift/post-6-2 6.2.4' The previous fix (#2059) only filtered out reconsolidation candidates whose name appeared as a fresh item. But this collision involved TWO reconsolidation candidates — the same installable appearing in two different old consolidated images (likely due to multiple consolidation passes over time). ## Fix After filtering fresh-vs-recon collisions, also deduplicate within the reconsolidation candidates themselves. When the same installable name appears in multiple old consolidated images, only the first occurrence is kept. The detection layer in consolidation.py (ValueError on duplicate subdir names) correctly caught and reported the issue; this PR fixes the prevention layer. ## Test New test `test_prepare_consolidation_items_rejects_duplicate_recon_names` reproduces the exact production scenario: two reconsolidation candidates from different consolidated images both with name 'compilers/swift/post-6-2 6.2.4'. 🤖 Generated by LLM (Claude, via OpenClaw)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
CEFS consolidation was failing with:
(from run https://github.com/compiler-explorer/infra/actions/runs/24232582644)
Root cause: A fresh item (
compilers/swift/static-sdk/sdk-0-1-0 6.2.4) and a reconsolidation item with the same name (pulled from an existing consolidated image) were both placed in the same consolidation group. Both sanitize to the same subdir name (compilers_swift_static-sdk_sdk-0-1-0_6.2.4). When parallel workers extracted them, the second found the directory already written by the first.Fix
Two-layer defence:
1.
cli/cefs.py(prevent): When assembling candidates, filter out reconsolidation items whose name already appears as a fresh item. The fresh item supersedes — it represents the current installed state; the reconsolidation item would just duplicate it.2.
consolidation.py(detect): Add a duplicate subdir_name guard inprepare_consolidation_itemsthat raisesValueErrorif two items in the same group would produce the same extraction directory. Safety net for any future code path that assembles a bad group.Test
New test
test_prepare_consolidation_items_rejects_duplicate_namesreproduces the exact production scenario: a fresh Swift static SDK item and a reconsolidation item with the same name in the same group.(I'm Molty, an AI assistant acting on behalf of @mattgodbolt)