Skip to content

feat: treat sbatch errors as job errors instead of workflow errors #322

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jul 20, 2025
Merged

feat: treat sbatch errors as job errors instead of workflow errors #322

Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
a0d47ef
fix: replace WorkflowError with report_job_error for sbatch failure
nikostr Jun 9, 2025
3170182
fix: wrap job in SubmittedJobInfo when calling report_job_error
nikostr Jun 9, 2025
3553b13
Add error message when reporting jobs failed due to bad sbatch call
nikostr Jun 9, 2025
87fb93c
Improve readability of error msg for bad sbatch jobs
nikostr Jun 9, 2025
8f1d7b6
Fix lints and format msg
nikostr Jul 4, 2025
e14cc36
Merge branch 'main' into handle-failed-sbatch
cmeesters Jul 4, 2025
d4082ba
Fix black formatting
nikostr Jul 6, 2025
2effd99
Merge branch 'main' into handle-failed-sbatch
cmeesters Jul 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 9 additions & 3 deletions snakemake_executor_plugin_slurm/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -366,9 +366,15 @@ def run_job(self, job: JobExecutorInterface):
process.returncode, call, output=err
)
except subprocess.CalledProcessError as e:
raise WorkflowError(
f"SLURM sbatch failed. The error message was {e.output}"
self.report_job_error(
SubmittedJobInfo(job),
msg=(
"SLURM sbatch failed. "
f"The error message was '{e.output.strip()}'.\n"
f" sbatch call:\n {call}\n"
),
)
return
# any other error message indicating failure?
if "submission failed" in err:
raise WorkflowError(
Expand Down Expand Up @@ -445,7 +451,7 @@ async def check_active_jobs(

# We use this sacct syntax for argument 'starttime' to keep it compatible
# with slurm < 20.11
sacct_starttime = f"{datetime.now() - timedelta(days = 2):%Y-%m-%dT%H:00}"
sacct_starttime = f"{datetime.now() - timedelta(days=2):%Y-%m-%dT%H:00}"
# previously we had
# f"--starttime now-2days --endtime now --name {self.run_uuid}"
# in line 218 - once v20.11 is definitively not in use any more,
Expand Down