Skip to content

AWS Batch: task polls forever when the job is missing from the DescribeJobs response — no bound, unlike grid's exitReadTimeout #7301

Description

@item084

Expected behavior and actual behavior

Expected: when an AWS Batch job's status can no longer be resolved — DescribeJobs keeps returning an empty result, or a result that does not contain the job — the task should eventually fail with a bounded, diagnosable error, the way the grid executors handle the analogous condition (job gone from squeue/qstat with no .exitcode file) via executor.exitReadTimeout.

Actual: the task polls forever. In AwsBatchTaskHandler, checkIfCompleted() calls describeJob(jobId); when the response is empty or the job is absent from it, describeJob() logs a DEBUG line and returns null, and checkIfCompleted() just returns false ("not done yet"):

final job = describeJob(jobId)
final done = job?.status() in [JobStatus.SUCCEEDED, JobStatus.FAILED]
if( done ) {
    ...
}
return false   // <- null falls through here, every poll, forever

There is no timestamp latch, no retry bound, and no mapping of "job persistently unresolvable" to a terminal state on this path (the only retry/backoff logic in the handler, resolveJobDefinition, covers job-definition resolution at submit time). The head process never exits, and the workflow hangs indefinitely.

This is easy to hit in practice because AWS Batch only retains job records for about 24 hours, and describe-jobs for an unknown/aged-out job id is a silent empty success, not an error:

$ aws batch describe-jobs --jobs 00000000-0000-0000-0000-000000000000
{
    "jobs": []
}
$ echo $?
0

So any job that outlives the retention window (or any intermittently-empty describe response) permanently wedges its task. The only externally visible signals are DEBUG-level lines in .nextflow.log, which makes the hang effectively undetectable for tooling that watches the head's stdout/exit code:

  • [AWS BATCH] cannot retrieve running status for job=<id> — describe response was empty;
  • [AWS BATCH] cannot find running status for job=<id> — response contained other (batched) jobs but not this one.

By contrast, GridTaskHandler.readExitStatus() latches exitTimestampMillis1 the moment the scheduler stops reporting the job as active, and fails the task once exitStatusReadTimeoutMillis elapses with no exit file — so the same underlying event (the substrate silently lost the job) is bounded on grid but unbounded on AWS Batch.

Suggested fix: mirror the grid pattern in AwsBatchTaskHandler.checkIfCompleted() — latch a timestamp on the first null from describeJob(), reset it whenever a describe succeeds, and fail the task (exit Integer.MAX_VALUE + a process error) once the configured exitReadTimeout elapses. Reusing executor.config.getExitReadTimeout(...) would keep the knob symmetric with the grid executors.

Steps to reproduce the problem

Fully reproducible locally with no AWS account: point the AWS SDK at a small fake Batch API (the SDK honors the AWS_ENDPOINT_URL_BATCH env var) and at s3proxy for the S3 work dir. The fake API implements RegisterJobDefinition/SubmitJob/DescribeJobs; after N seconds it starts returning {"jobs": []} for the submitted job — the exact shape real AWS returns for an aged-out job. The pipeline is a single sleep 3600 process on the awsbatch executor.

All four files are below; save them into one directory and run ./run.sh 15 (requires only Docker).

run.sh — orchestrates the whole repro
#!/usr/bin/env bash
# Reproduce the Nextflow nf-amazon "lost task polls forever" bug
# locally: a real nf-amazon@3.4.4 head against a fake Batch API that vanishes
# a job (empty DescribeJobs response), plus s3proxy for the work dir.
#
# Usage: ./run.sh [vanish_after_seconds] [pipeline]
#   pipeline: lost_task.nf (default, single task) | lost_task_ordered.nf
#             (two concurrent tasks — one vanishes, one stays healthy;
#              triggers the `cannot find running status` marker variant)
#
# No real AWS account or credentials needed. Requires Docker.
set -euo pipefail
cd "$(dirname "${BASH_SOURCE[0]}")"

VANISH_AFTER_S="${1:-15}"
PIPELINE="${2:-lost_task.nf}"
NETWORK=nf-lost-job-repro

cleanup() {
    docker rm -f spike-nextflow spike-fake-batch spike-s3proxy >/dev/null 2>&1 || true
    docker network rm "$NETWORK" >/dev/null 2>&1 || true
}
trap cleanup EXIT
cleanup
rm -rf work .nextflow* nxf-home 2>/dev/null || true

docker network create "$NETWORK" >/dev/null

docker run -d --name spike-s3proxy --network "$NETWORK" \
    -e S3PROXY_ENDPOINT=http://0.0.0.0:8080 \
    -e S3PROXY_AUTHORIZATION=none \
    -e S3PROXY_IDENTITY=test -e S3PROXY_CREDENTIAL=test \
    andrewgaul/s3proxy:s3proxy-2.9.0 >/dev/null
sleep 2
docker run --rm --network "$NETWORK" \
    -e AWS_ACCESS_KEY_ID=test -e AWS_SECRET_ACCESS_KEY=test -e AWS_DEFAULT_REGION=us-east-1 \
    amazon/aws-cli:2.15.0 --endpoint-url http://spike-s3proxy:8080 s3 mb s3://spike-bucket >/dev/null

docker run -d --name spike-fake-batch --network "$NETWORK" \
    -v "$(pwd)/fake_batch.py:/fake_batch.py:ro" \
    -e "VANISH_AFTER_S=${VANISH_AFTER_S}" \
    python:3.12-slim python3 /fake_batch.py

mkdir -p nxf-home
docker run --rm --network "$NETWORK" -v "$(pwd):/work" -w /work \
    -e NXF_HOME=/work/nxf-home \
    nextflow/nextflow:25.10.6 nextflow plugin install nf-amazon@3.4.4

echo "--- launching real Nextflow against the fake Batch endpoint ---"
docker run -d --name spike-nextflow --network "$NETWORK" \
    -v "$(pwd):/work" -w /work \
    -e NXF_HOME=/work/nxf-home \
    -e AWS_ENDPOINT_URL_BATCH=http://spike-fake-batch:8080 \
    -e AWS_ACCESS_KEY_ID=test -e AWS_SECRET_ACCESS_KEY=test -e AWS_REGION=us-east-1 \
    -e AWS_REQUEST_CHECKSUM_CALCULATION=when_required \
    -e AWS_RESPONSE_CHECKSUM_VALIDATION=when_required \
    -e NXF_ANSI_LOG=false \
    nextflow/nextflow:25.10.6 \
    nextflow run "$PIPELINE" -c nextflow.config >/dev/null

WATCH_S=$((VANISH_AFTER_S + 30))
echo "watching for ${WATCH_S}s past submit — expect the job to vanish at ~${VANISH_AFTER_S}s" \
     "and then poll forever with no ProcessFailedException..."
sleep "$WATCH_S"

echo "--- fake-batch log (tail) ---"
docker logs spike-fake-batch 2>&1 | tail -10
echo "--- nextflow container still running? ---"
docker ps --filter name=spike-nextflow --format "{{.Names}} {{.Status}}"
echo "--- unresolved-status markers seen ---"
grep -o "cannot retrieve running status\|cannot find running status" .nextflow.log 2>/dev/null | sort | uniq -c || true
echo "--- any terminal error? (should be none — that's the point) ---"
grep -i "ProcessFailedException" .nextflow.log 2>/dev/null || echo "(none — confirms poll-forever)"
fake_batch.py — minimal fake AWS Batch API
#!/usr/bin/env python3
"""Minimal fake AWS Batch API (rest-json, POST /v1/<op>) to reproduce a Nextflow nf-amazon poll-forever bug.

Implements just enough of RegisterJobDefinition / DescribeJobDefinitions /
SubmitJob / DescribeJobs for a real nf-amazon@3.4.4 AwsBatchTaskHandler to
submit a job successfully, then simulates a job aging out: after
VANISH_AFTER_S seconds since submit, DescribeJobs returns an empty `jobs`
list for that job id forever (the exact `{"jobs": []}` shape real AWS
returns for an aged-out job id).
"""
import json
import os
import time
import uuid
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

VANISH_AFTER_S = float(os.environ.get("VANISH_AFTER_S", "15"))
PORT = int(os.environ.get("PORT", "8080"))

_jobs: dict[str, float] = {}  # jobId -> submitted_at
_names: dict[str, str] = {}  # jobId -> jobName


class Handler(BaseHTTPRequestHandler):
    def _read_json(self) -> dict:
        length = int(self.headers.get("Content-Length", 0))
        raw = self.rfile.read(length) if length else b"{}"
        try:
            return json.loads(raw or b"{}")
        except json.JSONDecodeError:
            return {}

    def _respond(self, status: int, body: dict) -> None:
        payload = json.dumps(body).encode("utf-8")
        self.send_response(status)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(payload)))
        self.end_headers()
        self.wfile.write(payload)

    def log_message(self, fmt, *args):  # noqa: A002 - keep stdlib signature
        print(f"[fake-batch] {self.address_string()} {fmt % args}", flush=True)

    def do_POST(self) -> None:  # noqa: N802 - stdlib method name
        body = self._read_json()
        path = self.path

        if path == "/v1/describejobdefinitions":
            # Always empty -> forces the handler to register a fresh definition.
            self._respond(200, {"jobDefinitions": []})
            return

        if path == "/v1/registerjobdefinition":
            name = body.get("jobDefinitionName", "fake-job-def")
            self._respond(200, {
                "jobDefinitionName": name,
                "jobDefinitionArn": f"arn:aws:batch:us-east-1:000000000000:job-definition/{name}:1",
                "revision": 1,
            })
            return

        if path == "/v1/submitjob":
            job_id = str(uuid.uuid4())
            _jobs[job_id] = time.monotonic()
            name = body.get("jobName", "fake-job")
            _names[job_id] = name
            print(f"[fake-batch] SubmitJob -> jobId={job_id} name={name}", flush=True)
            self._respond(200, {
                "jobArn": f"arn:aws:batch:us-east-1:000000000000:job/{job_id}",
                "jobName": name,
                "jobId": job_id,
            })
            return

        if path == "/v1/describejobs":
            ids = body.get("jobs", [])
            jobs = []
            for job_id in ids:
                submitted_at = _jobs.get(job_id)
                if submitted_at is None:
                    continue  # unknown id -> also omitted, same as "vanished"
                name = _names.get(job_id, "")
                upper = name.upper()
                age = time.monotonic() - submitted_at
                # Only jobs named *LOST* ever vanish; others (e.g. a batched
                # sibling) stay RUNNING forever, so a batched describe can
                # come back non-empty while still missing the lost job.
                if "LOST" in upper and age >= VANISH_AFTER_S:
                    print(f"[fake-batch] DescribeJobs jobId={job_id} name={name} age={age:.1f}s "
                          f"-> VANISHED (omitted from response)", flush=True)
                    continue  # <-- the trigger: job silently absent from the response
                print(f"[fake-batch] DescribeJobs jobId={job_id} name={name} age={age:.1f}s -> RUNNING", flush=True)
                jobs.append({
                    "jobArn": f"arn:aws:batch:us-east-1:000000000000:job/{job_id}",
                    "jobName": name or "fake-job",
                    "jobId": job_id,
                    "jobQueue": "fake-queue",
                    "status": "RUNNING",
                    "startedAt": int(submitted_at * 1000),
                    "jobDefinition": "fake-job-def",
                })
            print(f"[fake-batch] DescribeJobs requested={ids} -> "
                  f"returning {len(jobs)}/{len(ids)} job(s)", flush=True)
            self._respond(200, {"jobs": jobs})
            return

        if path == "/v1/canceljob" or path == "/v1/terminatejob":
            self._respond(200, {})
            return

        print(f"[fake-batch] UNHANDLED {path} body={body}", flush=True)
        self._respond(200, {})


if __name__ == "__main__":
    print(f"[fake-batch] listening on 0.0.0.0:{PORT}, VANISH_AFTER_S={VANISH_AFTER_S}", flush=True)
    ThreadingHTTPServer(("0.0.0.0", PORT), Handler).serve_forever()
nextflow.config
plugins {
    id 'nf-amazon@3.4.4'
}

aws {
    region = 'us-east-1'
    accessKey = 'test'
    secretKey = 'test'
    client {
        endpoint = 'http://spike-s3proxy:8080'
        s3PathStyleAccess = true
    }
    batch {
        // No real cli/fusion needed — Batch task submission is intercepted at
        // the API layer, tasks never actually run on real infra.
        maxTransferAttempts = 1
    }
}

process {
    executor = 'awsbatch'
    queue = 'fake-queue'
    maxRetries = 0
    errorStrategy = 'terminate'
}

workDir = 's3://spike-bucket/work'

executor {
    awsbatch {
        // Poll fast so the spike doesn't need to wait ~ default cadence.
        pollInterval = '5s'
    }
}
lost_task.nf
#!/usr/bin/env nextflow
nextflow.enable.dsl=2

process LOST_TASK {
    executor 'awsbatch'
    container 'busybox'
    errorStrategy 'terminate'

    output:
    stdout

    script:
    """
    sleep 3600
    """
}

workflow {
    LOST_TASK()
}

Result: the head keeps polling the vanished job on every poll interval, indefinitely (observed for 10+ minutes; the same behavior holds as long as you care to wait), with no error raised and no exit. The same setup with two concurrent tasks (one vanishing, one healthy sibling that stays in the batched DescribeJobs response) reproduces the second marker variant, cannot find running status — also polling forever.

Program output

The head's stdout shows a normal submission and then nothing further:

N E X T F L O W  ~  version 25.10.6
Launching `lost_task.nf` [modest_lavoisier] DSL2 - revision: bfbc0efe34
[16/576e43] Submitted process > LOST_TASK

.nextflow.log (DEBUG) shows the unresolvable status repeating every poll, with no terminal error ever raised:

Jul-03 19:22:24.038 [AWSBatch-executor-1] DEBUG n.c.aws.batch.AwsBatchTaskHandler - [AWS BATCH] Process `LOST_TASK` submitted > job=9b27507c-9728-4292-8030-98c6bc4109f8; work-dir=s3://spike-bucket/work/16/576e43578876036ba70dbae89554fe
Jul-03 19:22:43.765 [Task monitor] DEBUG n.c.aws.batch.AwsBatchTaskHandler - [AWS BATCH] cannot retrieve running status for job=9b27507c-9728-4292-8030-98c6bc4109f8
Jul-03 19:22:53.765 [Task monitor] DEBUG n.c.aws.batch.AwsBatchTaskHandler - [AWS BATCH] cannot retrieve running status for job=9b27507c-9728-4292-8030-98c6bc4109f8
Jul-03 19:23:03.767 [Task monitor] DEBUG n.c.aws.batch.AwsBatchTaskHandler - [AWS BATCH] cannot retrieve running status for job=9b27507c-9728-4292-8030-98c6bc4109f8
... (repeats every ~10s indefinitely; observed 10+ minutes with zero errors and the JVM alive throughout)

Environment

  • Nextflow version: 25.10.6 (nf-amazon@3.4.4); the unbounded null path is still present on current master
  • Java version: OpenJDK 21 (Amazon Corretto, from the nextflow/nextflow:25.10.6 image)
  • Operating system: Linux
  • Bash version: GNU bash 5.2.21

Additional context

The trigger condition was confirmed against real AWS (empty describe-jobs success for an unknown job id, shown above); the poll-forever response to it was confirmed both by reading AwsBatchTaskHandler (25.10.6 binary and current master) and by the local end-to-end reproduction. A time directive does not help: it maps to Batch's attemptDurationSeconds, which AWS enforces on a job it still knows about — a vanished job has no record left for the timeout to act on.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions