fix: remove_dir now correctly removes symlinks (CEFS nightly cleanup)#2027
Merged
mattgodbolt merged 1 commit intomainfrom Mar 12, 2026
Merged
fix: remove_dir now correctly removes symlinks (CEFS nightly cleanup)#2027mattgodbolt merged 1 commit intomainfrom
mattgodbolt merged 1 commit intomainfrom
Conversation
On CEFS, nightly compiler paths (e.g. gcc-trunk-20250910) are symlinks pointing to /cefs/hash_name squashfs mounts rather than real directories. shutil.rmtree() on a symlink raises NotADirectoryError in Python 3.12+, which was silently swallowed by ignore_errors=True, leaving old symlinks to accumulate indefinitely (observed from Sep 10 2025 onwards). This prevented the CEFS GC from collecting the underlying images since they remained referenced, causing unbounded growth in CEFS storage. Fix: check for symlinks first and use unlink() rather than rmtree(). 🤖 Generated by LLM (Claude, via OpenClaw)
mattgodbolt
approved these changes
Mar 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On CEFS, nightly compiler paths (e.g.
gcc-trunk-20250910) are symlinks pointing to/cefs/<hash>_gcc-trunk-YYYYMMDDsquashfs mounts rather than real directories.NightlyInstallable.install()callsremove_dir()to clean up old versions (keeping onlynum_to_keep). Butremove_dir()was usingshutil.rmtree(..., ignore_errors=True). In Python 3.12+,shutil.rmtreeraisesNotADirectoryErrorwhen given a symlink — and withignore_errors=Truethis is silently swallowed. The old symlinks are never removed.This has been accumulating since ~Sep 10 2025 (when CEFS was first enabled for gcc-trunk nightly installs on the admin node), leaving months of daily gcc-trunk symlinks still present. Because those symlinks remain, the CEFS GC cannot collect the underlying squashfs images — causing unbounded growth in CEFS storage.
Fix
In
remove_dir(), check whether the path is a symlink first. If so, callunlink()instead ofrmtree(). The underlying CEFS content is left intact (as expected — we only want to remove the/opt/compiler-explorer/gcc-trunk-YYYYMMDDsymlink, not the squashfs image).Tests
Three new tests added to
bin/test/installation_test.py:All 552 existing tests pass.
(I'm Molty, an AI assistant acting on behalf of @mattgodbolt)