Skip to content

Commit 5cdcc02

Browse files
authored
Stop the backend suites logging captured output (#22581)
Since #22246 moved these suites to OSDC, `arm_ethos_u55 / models` and `arm_ethos_u85 / models` have died partway through reporting their failures. The cause is the runner agent's heap, not the pod: the log ends with `##[error]Exception of type 'System.OutOfMemoryException' was thrown.` Each Ethos-U failure captures a few hundred thousand lines of Vela operator listings, and the agent cannot process a step that size. `--show-capture=no` drops those sections from the terminal report. Verified on the default label and worker count: https://github.com/pytorch/executorch/actions/runs/33935989761 ``` success arm_ethos_u85, models <- never passed on OSDC before success arm_ethos_u55, models 9 failed, 13 passed, 20 skipped log: 12,596 lines <- was 458k, truncated where the agent died ``` Nothing is lost. `--show-capture` only affects what the terminal prints, so the captured streams still ship in the JSON report artifact (19MB, against 17MB before), and each failure keeps its traceback and exception message — `RuntimeError: Corstone simulation failed` and the arena sizes are still in the log. This applies to every backend suite, Linux and macOS, since `_test_backend.yml` sources the script in both jobs. Only the Ethos-U cells produce output at this scale today, but any of them could. The failures themselves are pre-existing and unrelated; the job was green with them before because `test_backend.sh` exits with the summary generator's status rather than pytest's. Authored with Claude Code. cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani
1 parent c570b6c commit 5cdcc02

1 file changed

Lines changed: 6 additions & 1 deletion

File tree

.ci/scripts/test_backend.sh

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -158,7 +158,12 @@ GOLDEN_DIR="${ARTIFACT_DIR}/golden-artifacts"
158158
export GOLDEN_ARTIFACTS_DIR="${GOLDEN_DIR}"
159159

160160
EXIT_CODE=0
161-
PYTEST_ARGS=(-c /dev/null -n auto)
161+
# An Ethos-U failure captures a few hundred thousand lines of Vela operator
162+
# listings, and the runner agent throws System.OutOfMemoryException processing
163+
# a step that size, taking the whole job down before pytest can report. The
164+
# reason for each failure is in its exception message and traceback, which are
165+
# unaffected.
166+
PYTEST_ARGS=(-c /dev/null -n auto --show-capture=no)
162167
if [[ ${#PYTEST_RETRY_ARGS[@]} -gt 0 ]]; then
163168
PYTEST_ARGS+=("${PYTEST_RETRY_ARGS[@]}")
164169
fi

0 commit comments

Comments
 (0)