The Windows x64 suite step loses its runner whenever the FTP login test runs

[xroche]

Adding tests/230_local-ftp-userpass.test in #1035 kills the x64 leg of windows-build every time it runs. The Win32 leg of the same commits stays green.

commit	test present	x64	Win32
186476ca	no	success	success
d6f4d71a	yes	failure (twice, the second an explicit re-run of the same sha)	success
6547eaca	yes	failure	success

The three x64 failures look identical. Build is green, Run the engine test suite (offline tests) sits at in_progress well past its 45-minute timeout-minutes, the if: always() upload steps never start, the job log blob comes back BlobNotFound, and the only annotation left is "The hosted runner lost communication with the server. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error." A healthy x64 run finishes that step in 13m48s. Win32's suite time moved by 18 seconds when the test landed, and 230 itself takes 9s there.

Nothing separates the two matrix legs but matrix.platform: same windows-2022 image, same steps, same 45-minute budget, same invocation. What differs is the binary.

None of the suite's own guards fired, which is what says this is more than a slow test. Every test runs under HTTRACK_TEST_TIMEOUT=600 bash ./test-timeout.sh, ci-windows-suite.sh checks a 1500s suite deadline between tests, and ci_suite_heartbeat kills the step after 900s without a progress line. A test that merely hung would have been reaped and named. All three are shell loops, so their silence says the machine had stopped scheduling them. It is also why this is not the base-rate wedge of #795: 0/3 on one platform against 3/3 on the other is not a 4-6% flake.

230's crawl flags are byte for byte those of 221_local-ftp-ctrlchars.test, which is green on Windows at 3s. The delta is --require-pass: 230 is the only test in the tree that makes the fixture answer USER with 331, and therefore the only one that ever reaches the PASS branch at src/htsftp.c:401-408. Every other FTP test gets "200 ok" to USER and skips PASS entirely. That branch runs in 230's first crawl, the one with a 6-byte user, so the cheap bisect is to keep only bob:secret and drop --require-pass, rather than to go on varying the credential length.

An audit of the rest of that path came back clean. jump_identification always yields end > src, so the new (size_t)(end - src) - 1 cannot underflow. The early return at src/htsftp.c:261 sets the same two fields as the pre-connect returns already sitting beside it at :285 and :303. STATUSCODE_INVALID is absent from can_retry (htsparse.c:3757), so no link loops. src/htsftp.c makes no allocation at all. The credentials reach neither the mirror filename nor the hash key, both of which go through jump_identification_const, so the #133 long-name shortening block is never entered. long is 32-bit on both MSVC targets and the tree holds a single _WIN64 conditional, the T_SOC typedef. The same tests are green on Linux x86-64 under ASan and UBSan with halt_on_error=1, on macOS arm64, and on Linux -m32, which is a genuine ILP32 control for this function.

So no mechanism is pinned, and the tempting inference does not hold: the wedge's shape carries no platform signal. Across the last 300 windows-build runs, 20 jobs blew past 45 minutes, split exactly 10 x64 and 10 Win32. What sets this branch apart is not that x64 is the fragile leg but that the correlation is far too strong to be that flake: at a 4% per-job base rate, 3-for-3 against 0-for-3 is around 1e-4. On Linux the same crawl peaks at 26 MB and the whole test runs in 7 seconds.

The nearest thing to a candidate is #1039. run_launch_ftp hardcodes a 300s control-channel timeout and ignores --timeout and --max-time, and back_wait exempts FTP slots from the mirror-limit abort, so one unanswered reply pins a crawl for five minutes. 230 drives four crawls under a 60-second harness cap.

Getting evidence is the awkward part, because a wedged job uploads nothing: tests/*.log and suite-progress.log reach the artifact only when the step ends on its own terms, and a Win32 artifact says nothing about x64. Two harness defects make that worse, both in the #795 family and worth fixing first:

• ci_suite_heartbeat runs its sleep and its now=$(hb_now) unguarded under the inherited set -e. MSYS fork emulation is exactly what starts failing under memory pressure, so the watchdog exits silently the first time it cannot fork, having written nothing. 171_watchdog-heartbeat.test stubs sleep as a shell function, so no test can see it.
• test-timeout.sh's poll loop sets only -u, so a sleep that cannot fork does not end the loop. It spins retrying the fork instead, amplifying the starvation it exists to bound.

Writing progress into $GITHUB_STEP_SUMMARY as the suite runs would also survive a lost runner, where an end-of-job artifact does not.

Until then #1035 skips the test on Windows. Its wire-level assertions are what prove the credentials reach the control channel unclipped, and they keep running on Linux and macOS.

[xroche]

Still open, but the picture has moved since the skip landed.

The candidate this issue named as nearest, #1039 (run_launch_ftp hardcoding a 300s control-channel read and ignoring --timeout/--max-time), was fixed in 7c6e5c3. #1055, where the engine freed the transfer slots while an FTP thread was still writing to one, landed too. Both are after b2cd2a5, which is where 230_local-ftp-userpass.test arrived already skipped on Windows, so neither fix has ever been tested against the wedge.

Four crawls under a 60s harness cap against a 300s unbounded read is a plausible whole explanation for a step that runs past 45 minutes and takes the runner with it, and that arithmetic no longer applies.

The cheap next step is to drop the is_windows gate and the expected_skips entry on a throwaway branch and read the x64 leg once. A wedge uploads nothing, so the run either goes green or costs one lost runner and no information, which is the same price as today. Not worth doing on master.