Request interrupted at CountDownLatch.await() returns 503; cause unknown

[gary357]

A request that completes without committing its response hangs the Lambda invocation until the runtime kills it, and the client receives a 503. Observed six times in a single 42-minute test run against a Lambda deployment.

Symptom

API Gateway returns 503. The Lambda log records:

FuseLess: StreamLambdaHandler: Request interrupted by Lambda runtime :: null
java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(Unknown Source)
java.base/java.util.concurrent.CountDownLatch.await(Unknown Source)
com.amazonaws.serverless.proxy.internal.LambdaContainerHandler.proxy(LambdaContainerHandler.java:217)
com.amazonaws.serverless.proxy.internal.LambdaContainerHandler.proxyStream(LambdaContainerHandler.java:258)
com.foundeo.fuseless.StreamLambdaHandler.handleRequest(StreamLambdaHandler.java:156)
com.amazonaws.services.lambda.runtime.api.client.EventHandlerLoader$2.call(EventHandlerLoader.java:604)

All six occurrences were byte-identical. Six different paths, both GET and POST, spread across 41 minutes.

Mechanism

LambdaContainerHandler.proxy() blocks on latch.await() until the servlet response is committed. In aws-serverless-java-container-core, countDown() has exactly one call site — inside AwsHttpServletResponse.flushBuffer():

public void flushBuffer() throws IOException {
    ...
    isCommitted = true;
    writersCountDownLatch.countDown();
}

If flushBuffer() is never reached, the latch is never released and the invocation blocks until the Lambda runtime interrupts it.

The library guards against this in AwsLambdaServletContainerHandler.doFilter():

protected void doFilter(HttpServletRequest request, HttpServletResponse response, Servlet servlet) {
    FilterChain chain = getFilterChain(request, servlet);
    chain.doFilter(request, response);
    if (requiresAsyncReDispatch(request)) { … }
    // if for some reason the response wasn't flushed yet, we force it here unless it's being processed asynchronously (WebFlux)
    if (!response.isCommitted() && request.getDispatcherType() != DispatcherType.ASYNC) {
        response.flushBuffer();
    }
}

FuseLess does not go through doFilter(). CFMLLambdaContainerHandler.handleRequest() calls the servlet directly:

StreamLambdaHandler.getCFMLServlet().service(req, httpServletResponse);

That is reasonable — FuseLess is a CFML container with no servlet filters to apply, and the commit guarantee is bundled inside filter machinery it has no reason to invoke. But it means nothing in the call path forces the flush when Lucee returns without committing.

Confirmed by grep: isCommitted, flushBuffer, startAsync and DispatcherType appear nowhere in the FuseLess source.

This is longstanding, not a regression — the service() call is present in the initial commit (2018-11-27) and unchanged since.

Related upstream fix

aws/serverless-java-container PR 1600, shipped in 3.0.1, fixes the same class of failure in their springboot4 module. Their description:

no message converter runs, no flush ever happens, countDown() is never called, and latch.await(15, TimeUnit.MINUTES) blocks indefinitely

Different trigger (an unconditional startAsync() skipping the same safety net), same mechanism. Their fix restored the path to the guard; FuseLess never had it.

Proposed change

Add the commit check after the servlet call in CFMLLambdaContainerHandler.handleRequest():

StreamLambdaHandler.getCFMLServlet().service(req, httpServletResponse);

if (!httpServletResponse.isCommitted()) {
    StreamLambdaHandler.log("Response not committed after service(); forcing flush. path=" + req.getRequestURI());
    httpServletResponse.flushBuffer();
}

Two parts, deliberately together:

1. The flush converts a hung invocation into a served response.
2. The log line records each occurrence. Without it the fault becomes silent rather than fixed, and the underlying condition — currently unidentified — would stop being observable.

Restoring doFilter() is not proposed: skipping the filter chain appears intentional and the guard can be replicated without it.

What this does not address

Why Lucee sometimes returns without committing the response is unknown. This makes the omission non-fatal; it does not explain it.

The failing requests were indistinguishable from successful ones in every logged respect — same path, same warm container, ~22ms duration against a 17–29ms neighbourhood, memory flat at 789 MB of 3008 MB, no exception logged, no application error. In one case the failure was roughly the 21st consecutive request to the same URL in the same container, after twenty successes.

Also worth noting: a forced flush may commit an empty or partial response, so the client could receive a blank 200 instead of a 503. Better for availability, and quieter — which is the reason for the log line.

Environment

• aws-serverless-java-container-core 2.1.5
• Java 21 Lambda runtime, SnapStart enabled
• Failures ruled out as SnapStart-related: each occurred 5–19 minutes after the nearest RESTORE_REPORT or INIT_REPORT
• Ruled out as idle-timeout-related: traffic was continuous, 1.6–2.3s between requests

[gary357]

Correction: the proposed guard was implemented, deployed, and did not fire — the uncommitted-response hypothesis is refuted

The fix proposed above was built and tested. It does not work, and the mechanism this issue describes is not what is happening. Recording that here so nobody implements it on the strength of the original write-up.

What was deployed

The commit guard, placed in the finally block of CFMLLambdaContainerHandler.handleRequest() so it would run whether the servlet returned normally or threw:

try {
    if (!httpServletResponse.isCommitted()) {
        StreamLambdaHandler.log("Response not committed after service(); forcing flush. path=" + req.getRequestURI());
        httpServletResponse.flushBuffer();
    }
} catch (Throwable flushError) {
    StreamLambdaHandler.log("Failed to force flush of uncommitted response: ", flushError);
}

Verified present in the deployed artifact by decompiling the Lambda package with javap — isCommitted(), the log string, and flushBuffer() all appear in CFMLLambdaContainerHandler.class as shipped, in all three exit paths the compiler duplicates a finally across.

Result

A full test suite run of ~3,450 requests against the patched build:

	before the guard	with the guard
Requests	3,615	3,448
503s	6	1
Request interrupted log lines	6	1
Guard fires (Response not committed)	n/a	0

The guard logged nothing, and a 503 with the identical signature occurred anyway. Same stack, same interrupt at CountDownLatch.await(), 28ms duration.

The drop from 6 to 1 is not attributable to the guard — it never executed its logging branch. One run against one run, on a fault occurring at roughly 0.17% of requests, is well within variance.

What this rules out

The guard logs unconditionally before flushing, so zero log lines across a run that produced an interrupt means either:

1. isCommitted() returned true — the response was committed, and something else prevented the latch from releasing; or
2. the finally block never ran — the interrupt struck while the request was still in flight, so handleRequest() never returned at all.

(2) fits the evidence. The stack trace shows the blocked thread inside LambdaContainerHandler.proxy(), which is the caller of handleRequest(). The proxy thread waits on the latch while a separate thread runs the servlet. If the runtime interrupts the waiting thread, the worker executing handleRequest() is abandoned mid-execution and its finally never runs.

So the guard is at the wrong layer. It can only act after the servlet call returns; this failure interrupts a thread that is still waiting for that to happen. No guard placed inside handleRequest can help.

What else has been eliminated since

Working from seven occurrences with gateway and Lambda evidence correlated on both sides:

candidate cause	how ruled out
SnapStart restore	every occurrence 5–19 min from the nearest RESTORE_REPORT / INIT_REPORT
Idle timeout	traffic continuous, 1.6–2.3s between requests
Memory pressure	flat 789–883 MB of 3008 MB across the surrounding sequence
Slow requests	the failing request sat at the 82nd percentile of latency; p99 is 12× slower and succeeds
Application exception	no exception logged, no error from the CFML layer
Uncommitted response	this comment
Container recycling	every affected container served 284–2,619 more requests afterwards, surviving 145–940s longer

The container-recycling result also confirms an observation already recorded in FuseLessExceptionHandler's Javadoc — the interrupt is "a transient signal (not a container shutdown — the next request on the same container succeeds normally)."

Current state of the problem

Something interrupts the Lambda runtime thread mid-invocation, on a request of unremarkable duration doing unremarkable work, inside a healthy container that continues serving normally afterwards, at roughly 1 in 1,000 to 1 in 3,500 requests under sustained load.

No replacement hypothesis is offered. Two were tested in sequence and both failed. Proposing a third on the current evidence would be guessing.

The latent bug in the original report still stands

One thing from the original write-up is unaffected and remains worth fixing on its own merits: handleRequest catches Throwable, logs, and returns normally, leaving the response uncommitted. That path would produce this hang. It is not what fired here — filtering the logs for Servlet Request Threw Exception across the affected runs returns zero events — but it will produce the same symptom the first time the CFML layer throws there.

That is a real defect independent of the fault this issue was opened about.