Expected behavior and actual behavior
Expected: when an AWS Batch job's status can no longer be resolved — DescribeJobs keeps returning an empty result, or a result that does not contain the job — the task should eventually fail with a bounded, diagnosable error, the way the grid executors handle the analogous condition (job gone from squeue/qstat with no .exitcode file) via executor.exitReadTimeout.
Actual: the task polls forever. In AwsBatchTaskHandler, checkIfCompleted() calls describeJob(jobId); when the response is empty or the job is absent from it, describeJob() logs a DEBUG line and returns null, and checkIfCompleted() just returns false ("not done yet"):
final job = describeJob(jobId)
final done = job?.status() in [JobStatus.SUCCEEDED, JobStatus.FAILED]
if( done ) {
...
}
return false // <- null falls through here, every poll, forever
There is no timestamp latch, no retry bound, and no mapping of "job persistently unresolvable" to a terminal state on this path (the only retry/backoff logic in the handler, resolveJobDefinition, covers job-definition resolution at submit time). The head process never exits, and the workflow hangs indefinitely.
This is easy to hit in practice because AWS Batch only retains job records for about 24 hours, and describe-jobs for an unknown/aged-out job id is a silent empty success, not an error:
$ aws batch describe-jobs --jobs 00000000-0000-0000-0000-000000000000
{
"jobs": []
}
$ echo $?
0
So any job that outlives the retention window (or any intermittently-empty describe response) permanently wedges its task. The only externally visible signals are DEBUG-level lines in .nextflow.log, which makes the hang effectively undetectable for tooling that watches the head's stdout/exit code:
[AWS BATCH] cannot retrieve running status for job=<id> — describe response was empty;
[AWS BATCH] cannot find running status for job=<id> — response contained other (batched) jobs but not this one.
By contrast, GridTaskHandler.readExitStatus() latches exitTimestampMillis1 the moment the scheduler stops reporting the job as active, and fails the task once exitStatusReadTimeoutMillis elapses with no exit file — so the same underlying event (the substrate silently lost the job) is bounded on grid but unbounded on AWS Batch.
Suggested fix: mirror the grid pattern in AwsBatchTaskHandler.checkIfCompleted() — latch a timestamp on the first null from describeJob(), reset it whenever a describe succeeds, and fail the task (exit Integer.MAX_VALUE + a process error) once the configured exitReadTimeout elapses. Reusing executor.config.getExitReadTimeout(...) would keep the knob symmetric with the grid executors.
Steps to reproduce the problem
Fully reproducible locally with no AWS account: point the AWS SDK at a small fake Batch API (the SDK honors the AWS_ENDPOINT_URL_BATCH env var) and at s3proxy for the S3 work dir. The fake API implements RegisterJobDefinition/SubmitJob/DescribeJobs; after N seconds it starts returning {"jobs": []} for the submitted job — the exact shape real AWS returns for an aged-out job. The pipeline is a single sleep 3600 process on the awsbatch executor.
All four files are below; save them into one directory and run ./run.sh 15 (requires only Docker).
run.sh — orchestrates the whole repro
#!/usr/bin/env bash
# Reproduce the Nextflow nf-amazon "lost task polls forever" bug
# locally: a real nf-amazon@3.4.4 head against a fake Batch API that vanishes
# a job (empty DescribeJobs response), plus s3proxy for the work dir.
#
# Usage: ./run.sh [vanish_after_seconds] [pipeline]
# pipeline: lost_task.nf (default, single task) | lost_task_ordered.nf
# (two concurrent tasks — one vanishes, one stays healthy;
# triggers the `cannot find running status` marker variant)
#
# No real AWS account or credentials needed. Requires Docker.
set -euo pipefail
cd "$(dirname "${BASH_SOURCE[0]}")"
VANISH_AFTER_S="${1:-15}"
PIPELINE="${2:-lost_task.nf}"
NETWORK=nf-lost-job-repro
cleanup() {
docker rm -f spike-nextflow spike-fake-batch spike-s3proxy >/dev/null 2>&1 || true
docker network rm "$NETWORK" >/dev/null 2>&1 || true
}
trap cleanup EXIT
cleanup
rm -rf work .nextflow* nxf-home 2>/dev/null || true
docker network create "$NETWORK" >/dev/null
docker run -d --name spike-s3proxy --network "$NETWORK" \
-e S3PROXY_ENDPOINT=http://0.0.0.0:8080 \
-e S3PROXY_AUTHORIZATION=none \
-e S3PROXY_IDENTITY=test -e S3PROXY_CREDENTIAL=test \
andrewgaul/s3proxy:s3proxy-2.9.0 >/dev/null
sleep 2
docker run --rm --network "$NETWORK" \
-e AWS_ACCESS_KEY_ID=test -e AWS_SECRET_ACCESS_KEY=test -e AWS_DEFAULT_REGION=us-east-1 \
amazon/aws-cli:2.15.0 --endpoint-url http://spike-s3proxy:8080 s3 mb s3://spike-bucket >/dev/null
docker run -d --name spike-fake-batch --network "$NETWORK" \
-v "$(pwd)/fake_batch.py:/fake_batch.py:ro" \
-e "VANISH_AFTER_S=${VANISH_AFTER_S}" \
python:3.12-slim python3 /fake_batch.py
mkdir -p nxf-home
docker run --rm --network "$NETWORK" -v "$(pwd):/work" -w /work \
-e NXF_HOME=/work/nxf-home \
nextflow/nextflow:25.10.6 nextflow plugin install nf-amazon@3.4.4
echo "--- launching real Nextflow against the fake Batch endpoint ---"
docker run -d --name spike-nextflow --network "$NETWORK" \
-v "$(pwd):/work" -w /work \
-e NXF_HOME=/work/nxf-home \
-e AWS_ENDPOINT_URL_BATCH=http://spike-fake-batch:8080 \
-e AWS_ACCESS_KEY_ID=test -e AWS_SECRET_ACCESS_KEY=test -e AWS_REGION=us-east-1 \
-e AWS_REQUEST_CHECKSUM_CALCULATION=when_required \
-e AWS_RESPONSE_CHECKSUM_VALIDATION=when_required \
-e NXF_ANSI_LOG=false \
nextflow/nextflow:25.10.6 \
nextflow run "$PIPELINE" -c nextflow.config >/dev/null
WATCH_S=$((VANISH_AFTER_S + 30))
echo "watching for ${WATCH_S}s past submit — expect the job to vanish at ~${VANISH_AFTER_S}s" \
"and then poll forever with no ProcessFailedException..."
sleep "$WATCH_S"
echo "--- fake-batch log (tail) ---"
docker logs spike-fake-batch 2>&1 | tail -10
echo "--- nextflow container still running? ---"
docker ps --filter name=spike-nextflow --format "{{.Names}} {{.Status}}"
echo "--- unresolved-status markers seen ---"
grep -o "cannot retrieve running status\|cannot find running status" .nextflow.log 2>/dev/null | sort | uniq -c || true
echo "--- any terminal error? (should be none — that's the point) ---"
grep -i "ProcessFailedException" .nextflow.log 2>/dev/null || echo "(none — confirms poll-forever)"
fake_batch.py — minimal fake AWS Batch API
#!/usr/bin/env python3
"""Minimal fake AWS Batch API (rest-json, POST /v1/<op>) to reproduce a Nextflow nf-amazon poll-forever bug.
Implements just enough of RegisterJobDefinition / DescribeJobDefinitions /
SubmitJob / DescribeJobs for a real nf-amazon@3.4.4 AwsBatchTaskHandler to
submit a job successfully, then simulates a job aging out: after
VANISH_AFTER_S seconds since submit, DescribeJobs returns an empty `jobs`
list for that job id forever (the exact `{"jobs": []}` shape real AWS
returns for an aged-out job id).
"""
import json
import os
import time
import uuid
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
VANISH_AFTER_S = float(os.environ.get("VANISH_AFTER_S", "15"))
PORT = int(os.environ.get("PORT", "8080"))
_jobs: dict[str, float] = {} # jobId -> submitted_at
_names: dict[str, str] = {} # jobId -> jobName
class Handler(BaseHTTPRequestHandler):
def _read_json(self) -> dict:
length = int(self.headers.get("Content-Length", 0))
raw = self.rfile.read(length) if length else b"{}"
try:
return json.loads(raw or b"{}")
except json.JSONDecodeError:
return {}
def _respond(self, status: int, body: dict) -> None:
payload = json.dumps(body).encode("utf-8")
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(payload)))
self.end_headers()
self.wfile.write(payload)
def log_message(self, fmt, *args): # noqa: A002 - keep stdlib signature
print(f"[fake-batch] {self.address_string()} {fmt % args}", flush=True)
def do_POST(self) -> None: # noqa: N802 - stdlib method name
body = self._read_json()
path = self.path
if path == "/v1/describejobdefinitions":
# Always empty -> forces the handler to register a fresh definition.
self._respond(200, {"jobDefinitions": []})
return
if path == "/v1/registerjobdefinition":
name = body.get("jobDefinitionName", "fake-job-def")
self._respond(200, {
"jobDefinitionName": name,
"jobDefinitionArn": f"arn:aws:batch:us-east-1:000000000000:job-definition/{name}:1",
"revision": 1,
})
return
if path == "/v1/submitjob":
job_id = str(uuid.uuid4())
_jobs[job_id] = time.monotonic()
name = body.get("jobName", "fake-job")
_names[job_id] = name
print(f"[fake-batch] SubmitJob -> jobId={job_id} name={name}", flush=True)
self._respond(200, {
"jobArn": f"arn:aws:batch:us-east-1:000000000000:job/{job_id}",
"jobName": name,
"jobId": job_id,
})
return
if path == "/v1/describejobs":
ids = body.get("jobs", [])
jobs = []
for job_id in ids:
submitted_at = _jobs.get(job_id)
if submitted_at is None:
continue # unknown id -> also omitted, same as "vanished"
name = _names.get(job_id, "")
upper = name.upper()
age = time.monotonic() - submitted_at
# Only jobs named *LOST* ever vanish; others (e.g. a batched
# sibling) stay RUNNING forever, so a batched describe can
# come back non-empty while still missing the lost job.
if "LOST" in upper and age >= VANISH_AFTER_S:
print(f"[fake-batch] DescribeJobs jobId={job_id} name={name} age={age:.1f}s "
f"-> VANISHED (omitted from response)", flush=True)
continue # <-- the trigger: job silently absent from the response
print(f"[fake-batch] DescribeJobs jobId={job_id} name={name} age={age:.1f}s -> RUNNING", flush=True)
jobs.append({
"jobArn": f"arn:aws:batch:us-east-1:000000000000:job/{job_id}",
"jobName": name or "fake-job",
"jobId": job_id,
"jobQueue": "fake-queue",
"status": "RUNNING",
"startedAt": int(submitted_at * 1000),
"jobDefinition": "fake-job-def",
})
print(f"[fake-batch] DescribeJobs requested={ids} -> "
f"returning {len(jobs)}/{len(ids)} job(s)", flush=True)
self._respond(200, {"jobs": jobs})
return
if path == "/v1/canceljob" or path == "/v1/terminatejob":
self._respond(200, {})
return
print(f"[fake-batch] UNHANDLED {path} body={body}", flush=True)
self._respond(200, {})
if __name__ == "__main__":
print(f"[fake-batch] listening on 0.0.0.0:{PORT}, VANISH_AFTER_S={VANISH_AFTER_S}", flush=True)
ThreadingHTTPServer(("0.0.0.0", PORT), Handler).serve_forever()
nextflow.config
plugins {
id 'nf-amazon@3.4.4'
}
aws {
region = 'us-east-1'
accessKey = 'test'
secretKey = 'test'
client {
endpoint = 'http://spike-s3proxy:8080'
s3PathStyleAccess = true
}
batch {
// No real cli/fusion needed — Batch task submission is intercepted at
// the API layer, tasks never actually run on real infra.
maxTransferAttempts = 1
}
}
process {
executor = 'awsbatch'
queue = 'fake-queue'
maxRetries = 0
errorStrategy = 'terminate'
}
workDir = 's3://spike-bucket/work'
executor {
awsbatch {
// Poll fast so the spike doesn't need to wait ~ default cadence.
pollInterval = '5s'
}
}
lost_task.nf
#!/usr/bin/env nextflow
nextflow.enable.dsl=2
process LOST_TASK {
executor 'awsbatch'
container 'busybox'
errorStrategy 'terminate'
output:
stdout
script:
"""
sleep 3600
"""
}
workflow {
LOST_TASK()
}
Result: the head keeps polling the vanished job on every poll interval, indefinitely (observed for 10+ minutes; the same behavior holds as long as you care to wait), with no error raised and no exit. The same setup with two concurrent tasks (one vanishing, one healthy sibling that stays in the batched DescribeJobs response) reproduces the second marker variant, cannot find running status — also polling forever.
Program output
The head's stdout shows a normal submission and then nothing further:
N E X T F L O W ~ version 25.10.6
Launching `lost_task.nf` [modest_lavoisier] DSL2 - revision: bfbc0efe34
[16/576e43] Submitted process > LOST_TASK
.nextflow.log (DEBUG) shows the unresolvable status repeating every poll, with no terminal error ever raised:
Jul-03 19:22:24.038 [AWSBatch-executor-1] DEBUG n.c.aws.batch.AwsBatchTaskHandler - [AWS BATCH] Process `LOST_TASK` submitted > job=9b27507c-9728-4292-8030-98c6bc4109f8; work-dir=s3://spike-bucket/work/16/576e43578876036ba70dbae89554fe
Jul-03 19:22:43.765 [Task monitor] DEBUG n.c.aws.batch.AwsBatchTaskHandler - [AWS BATCH] cannot retrieve running status for job=9b27507c-9728-4292-8030-98c6bc4109f8
Jul-03 19:22:53.765 [Task monitor] DEBUG n.c.aws.batch.AwsBatchTaskHandler - [AWS BATCH] cannot retrieve running status for job=9b27507c-9728-4292-8030-98c6bc4109f8
Jul-03 19:23:03.767 [Task monitor] DEBUG n.c.aws.batch.AwsBatchTaskHandler - [AWS BATCH] cannot retrieve running status for job=9b27507c-9728-4292-8030-98c6bc4109f8
... (repeats every ~10s indefinitely; observed 10+ minutes with zero errors and the JVM alive throughout)
Environment
- Nextflow version: 25.10.6 (
nf-amazon@3.4.4); the unbounded null path is still present on current master
- Java version: OpenJDK 21 (Amazon Corretto, from the
nextflow/nextflow:25.10.6 image)
- Operating system: Linux
- Bash version: GNU bash 5.2.21
Additional context
The trigger condition was confirmed against real AWS (empty describe-jobs success for an unknown job id, shown above); the poll-forever response to it was confirmed both by reading AwsBatchTaskHandler (25.10.6 binary and current master) and by the local end-to-end reproduction. A time directive does not help: it maps to Batch's attemptDurationSeconds, which AWS enforces on a job it still knows about — a vanished job has no record left for the timeout to act on.
Expected behavior and actual behavior
Expected: when an AWS Batch job's status can no longer be resolved —
DescribeJobskeeps returning an empty result, or a result that does not contain the job — the task should eventually fail with a bounded, diagnosable error, the way the grid executors handle the analogous condition (job gone fromsqueue/qstatwith no.exitcodefile) viaexecutor.exitReadTimeout.Actual: the task polls forever. In
AwsBatchTaskHandler,checkIfCompleted()callsdescribeJob(jobId); when the response is empty or the job is absent from it,describeJob()logs a DEBUG line and returnsnull, andcheckIfCompleted()just returnsfalse("not done yet"):There is no timestamp latch, no retry bound, and no mapping of "job persistently unresolvable" to a terminal state on this path (the only retry/backoff logic in the handler,
resolveJobDefinition, covers job-definition resolution at submit time). The head process never exits, and the workflow hangs indefinitely.This is easy to hit in practice because AWS Batch only retains job records for about 24 hours, and
describe-jobsfor an unknown/aged-out job id is a silent empty success, not an error:So any job that outlives the retention window (or any intermittently-empty describe response) permanently wedges its task. The only externally visible signals are DEBUG-level lines in
.nextflow.log, which makes the hang effectively undetectable for tooling that watches the head's stdout/exit code:[AWS BATCH] cannot retrieve running status for job=<id>— describe response was empty;[AWS BATCH] cannot find running status for job=<id>— response contained other (batched) jobs but not this one.By contrast,
GridTaskHandler.readExitStatus()latchesexitTimestampMillis1the moment the scheduler stops reporting the job as active, and fails the task onceexitStatusReadTimeoutMilliselapses with no exit file — so the same underlying event (the substrate silently lost the job) is bounded on grid but unbounded on AWS Batch.Suggested fix: mirror the grid pattern in
AwsBatchTaskHandler.checkIfCompleted()— latch a timestamp on the firstnullfromdescribeJob(), reset it whenever a describe succeeds, and fail the task (exitInteger.MAX_VALUE+ a process error) once the configuredexitReadTimeoutelapses. Reusingexecutor.config.getExitReadTimeout(...)would keep the knob symmetric with the grid executors.Steps to reproduce the problem
Fully reproducible locally with no AWS account: point the AWS SDK at a small fake Batch API (the SDK honors the
AWS_ENDPOINT_URL_BATCHenv var) and at s3proxy for the S3 work dir. The fake API implementsRegisterJobDefinition/SubmitJob/DescribeJobs; after N seconds it starts returning{"jobs": []}for the submitted job — the exact shape real AWS returns for an aged-out job. The pipeline is a singlesleep 3600process on theawsbatchexecutor.All four files are below; save them into one directory and run
./run.sh 15(requires only Docker).run.sh— orchestrates the whole reprofake_batch.py— minimal fake AWS Batch APInextflow.configplugins { id 'nf-amazon@3.4.4' } aws { region = 'us-east-1' accessKey = 'test' secretKey = 'test' client { endpoint = 'http://spike-s3proxy:8080' s3PathStyleAccess = true } batch { // No real cli/fusion needed — Batch task submission is intercepted at // the API layer, tasks never actually run on real infra. maxTransferAttempts = 1 } } process { executor = 'awsbatch' queue = 'fake-queue' maxRetries = 0 errorStrategy = 'terminate' } workDir = 's3://spike-bucket/work' executor { awsbatch { // Poll fast so the spike doesn't need to wait ~ default cadence. pollInterval = '5s' } }lost_task.nfResult: the head keeps polling the vanished job on every poll interval, indefinitely (observed for 10+ minutes; the same behavior holds as long as you care to wait), with no error raised and no exit. The same setup with two concurrent tasks (one vanishing, one healthy sibling that stays in the batched
DescribeJobsresponse) reproduces the second marker variant,cannot find running status— also polling forever.Program output
The head's stdout shows a normal submission and then nothing further:
.nextflow.log(DEBUG) shows the unresolvable status repeating every poll, with no terminal error ever raised:Environment
nf-amazon@3.4.4); the unbounded null path is still present on currentmasternextflow/nextflow:25.10.6image)Additional context
The trigger condition was confirmed against real AWS (empty
describe-jobssuccess for an unknown job id, shown above); the poll-forever response to it was confirmed both by readingAwsBatchTaskHandler(25.10.6 binary and current master) and by the local end-to-end reproduction. Atimedirective does not help: it maps to Batch'sattemptDurationSeconds, which AWS enforces on a job it still knows about — a vanished job has no record left for the timeout to act on.