misc: Keep distributed runs going after missing checkpoints

[jensen-yan]

Motivation

The weekly SMT SPEC06 1.0c job can fail before launching any GEM5 jobs when the distributed runner pre-resolves every selected checkpoint and one list entry is missing. In run 29038421818, the SMT checkpoint root was missing hmmer_nph3/0 and hmmer_retro/0, while the rest of the workload set was valid.

Approach

• Treat missing checkpoint preflight failures like other workload failures.
• Write abort/log.txt for the affected workload and continue scheduling the remaining workloads.
• Keep the distributed runner exit status at 0 so workflow check_result remains the authority for failing on aborts.

Validation

• PYTHONPYCACHEPREFIX=/tmp/gem5-dist-missing-pycache python3 -m py_compile util/xs_scripts/distributed_sim.py
• Fake local runner using /bin/true: one valid checkpoint completed, one missing checkpoint produced abort/log.txt, runner summary reported completed=1 failed=1 and exited 0.
• Confirmed the current SMT 1.0c checkpoint list/root is missing exactly hmmer_nph3_0 and hmmer_retro_0.

Summary by CodeRabbit

• Bug Fixes
  • Improved scheduler resilience when checkpoint files are missing or cannot be resolved.
  • Affected workloads are now marked as failed with clear status and log entries, while remaining workloads continue to be scheduled.
  • Standardized failure status updates and logging for launcher and checkpoint failures.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: a65011a3-3913-49ba-a57b-983655d74fdf

📥 Commits

Reviewing files that changed from the base of the PR and between 45bed11 and 3146f75.

📒 Files selected for processing (1)

• util/xs_scripts/distributed_sim.py

---

📝 Walkthrough

Walkthrough

Changes

Scheduler failure handling

Layer / File(s)	Summary
Shared workload failure markers util/xs_scripts/distributed_sim.py	Adds mark_work_dir_failure for updating status markers and logs, and routes launcher failures through it.
Fault-tolerant checkpoint resolution util/xs_scripts/distributed_sim.py	Handles checkpoint lookup failures per workload, records failure state, updates counters, and continues scheduling other workloads.

Estimated code review effort: 3 (Moderate) | ~15 minutes

Possibly related PRs

• OpenXiangShan/GEM5#936: Modifies the same distributed simulation launcher and scheduler entry points.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: continuing distributed runs when checkpoints are missing.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/distributed-skip-missing-checkpoints

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

Pull request overview

This PR updates the distributed simulation scheduler (util/xs_scripts/distributed_sim.py) so that missing checkpoints during the preflight “resolve all checkpoints” phase are treated as per-workload failures rather than hard-stopping the entire distributed run. This keeps weekly/CI-style distributed runs progressing when only a subset of selected checkpoints is missing.

Changes:

• Refactors failure marker/log writing into a reusable helper (mark_work_dir_failure) and reuses it for launcher failures.
• Catches FileNotFoundError from find_checkpoint() during preflight, writes failure markers/logs for that workload, increments the failure counter, and continues scheduling remaining workloads.
• Adjusts preflight scheduling list construction to only enqueue workloads that are either already completed (to be skipped later) or have a successfully resolved checkpoint.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

🚀 Coremark Smoke Test Results

Branch	IPC	Change
Base (xs-dev)	2.3162	-
This PR	2.3162	➡️ 0.0000 (0.00%)

✅ Difftest smoke test passed!