DelayedExecutionFlowImpl memory leak persists in 4.10.x — defuse() returns shared flow on keep-alive connections

[dualc]

Expected Behavior

DelayedExecutionFlowImpl memory leak persists in 4.10.x — defuse() returns shared flow on keep-alive connections

Expected Behavior

On a long-lived HTTP/1.1 keep-alive connection, each request should use an independent ExecutionFlow. Memory usage should remain stable regardless of how many requests are served on a single connection.

Actual Behavior

DelayedExecutionFlowImpl step nodes (Map, FlatMap, OnErrorResume, OnComplete) accumulate indefinitely on keep-alive connections, growing by ~8 steps per request until OOM. This is the same symptom reported in #10677.

Heap histogram from production (Micronaut 4.10.26, ~20 million requests served):

  59,821,130  io.micronaut.core.execution.DelayedExecutionFlowImpl$Map
  59,821,124  io.micronaut.core.execution.DelayedExecutionFlowImpl$OnErrorResume
  39,880,756  io.micronaut.core.execution.DelayedExecutionFlowImpl$FlatMap
  19,940,378  io.micronaut.core.execution.DelayedExecutionFlowImpl$OnComplete

Relationship to #10677 and PR #11546

PR #11546 added cleanup of DelayedExecutionFlowImpl internal state after completion. This helps when a flow completes normally. However, in the scenario described below, the shared flow never completes, so the cleanup never runs, and step nodes accumulate without bound.

Root Cause Analysis

The root cause is in ReactorExecutionFlowImpl.defuse() (source):

// ReactorExecutionFlowImpl.defuse()
if (publisher instanceof FlowAsMono<T> flowAsMono) {
    return flowAsMono.flow;  // <-- returns the SAME underlying flow every time
}

When a route handler returns a reactive type (e.g., Mono<HttpResponse>), Micronaut's request pipeline converts it through:

Mono → ReactorExecutionFlowImpl → (filter chain) → FlowAsMono → defuse() → underlying flow

The problem is that on a keep-alive connection, defuse() unwraps FlowAsMono and returns the same underlying DelayedExecutionFlowImpl from a previous request. This flow was created for the first request and never completed (the completion signal was consumed by the subscriber path, not the flow path).

Each subsequent request then appends its filter chain steps to this shared flow:

Request 1: flow → Map → OnErrorResume → Map → OnErrorResume → Map → OnComplete → Map → FlatMap
Request 2:      → Map → OnErrorResume → Map → OnErrorResume → Map → OnComplete → Map → FlatMap
Request 3:      → Map → OnErrorResume → ...
...
Request N: chain grows by ~8 nodes per request, never GC'd until connection closes

The responses are still written correctly through the Reactor subscriber path (FlowAsMono.subscribe()), so the server functions normally — it's a silent memory leak.

Proposed Fix

In ReactorExecutionFlowImpl.defuse(), when unwrapping a FlowAsMono, only return the underlying flow if it has already completed (immutable, safe to reuse). Otherwise, fall through to the subscriber path which creates an independent flow:

if (publisher instanceof FlowAsMono<T> flowAsMono) {
    ImperativeExecutionFlow<?> completed = flowAsMono.flow.tryComplete();
    if (completed != null) {
        return (ExecutionFlow<T>) completed;
    }
    // fall through to subscriber path — creates independent flow
}

This preserves the optimization for completed flows while preventing step accumulation on incomplete/shared flows.

Workaround

We added a Netty ChannelDuplexHandler that limits max requests per keep-alive connection (default 10,000). When the limit is reached, it adds Connection: close to the response, forcing the client to create a new connection and breaking the step accumulation chain.

Environment Information

• Micronaut: 4.10.26
• JDK: 21
• OS: Linux (Kubernetes)
• Application: Kestra fork, webserver mode
• Production traffic: ~78K req/s across multiple pods, keep-alive connections from nginx upstream

Actual Behaviour

No response

Steps To Reproduce

No response

Environment Information

No response

Example Application

No response

Version

4.10.26

[swaran21]

Thanks for the incredibly detailed root cause analysis, @dualc ! I'd like to pick this up and investigate the proposed fix.

My initial plan is to:

Update ReactorExecutionFlowImpl.defuse() so that an underlying FlowAsMono is only reused when its flow has completed, falling back to the subscriber path for incomplete flows.
Add an integration test that sends multiple requests over a single HTTP/1.1 keep-alive connection and verifies that execution flows are not shared between requests.
Verify that the existing behavior for completed flows is preserved.
I'll investigate the existing execution-flow tests and open a PR once I have a reproducer and validated fix.

[swaran21]

Update: I traced the unsafe sharing to the Reactor/ExecutionFlow conversion boundary and opened PR #12950.

ReactorExecutionFlowImpl.defuse() directly returned the wrapped flow for every FlowAsMono, including incomplete flows. A regression test that defuses the same incomplete FlowAsMono twice reproduces the problem: both calls return the same mutable underlying flow before this change.

The PR reuses the wrapped flow only when tryComplete() returns a completed imperative flow. Incomplete flows fall through to the existing subscriber path, which creates an independent delayed flow per conversion. It also verifies that the completed-flow fast path remains intact.

Validation was run with GraalVM CE 25.1.3 (JDK 25.0.3):

• ReactorExecutionFlowImplSpec: 11/11 passed (forced uncached run)
• :micronaut-http:check: passed (forced uncached run)

I attempted the repository-wide check on Windows as well; it reached micronaut-http-client but has two unrelated platform/environment failures in HttpHeadSpec (Windows path separator) and PcapLoggingSpec (no generated PCAP files).

A response-only HTTP/1.1 keep-alive test would pass before and after this change because responses are still written correctly while the internal step chain grows. The added regression test directly verifies the sharing invariant at the point where the bug occurs. If there is a minimal end-to-end reproducer for the retained chain, I can add that coverage too.

[yawkat]

Can you please provide a test case or some more info as to how you're hitting this issue? ExecutionFlow does not support multiple subscribers

[dualc]

Can you please provide a test case or some more info as to how you're hitting this issue? ExecutionFlow does not support multiple subscribers

I cannot reliably reproduce the issue either, but it occurs consistently in our production environment. Could you provide some troubleshooting suggestions? I have several dump files available for analysis.

[graemerocher]

Triage: likely valid, but still awaiting a maintainer-accepted reproducer. The current ReactorExecutionFlowImpl.defuse() still unwraps FlowAsMono directly, so the suspected sharing path remains present. This is related to #10677, but not a duplicate: the report identifies the incomplete-flow/keep-alive path, and PR #12950 supplied a focused unit reproducer for the unsafe sharing invariant before being closed without merge. The production heap evidence and the connection-close workaround also argue against closing as unreproducible.

Suggested next step: add an HTTP/1.1 keep-alive integration test that sends repeated requests through the same connection and asserts that each route conversion receives an independent incomplete flow, or attach a small heap/flight-recorder capture showing the retained DelayedExecutionFlowImpl chain. The proposed fix in #12950—reuse a FlowAsMono only when tryComplete() returns an immutable completed flow, otherwise use the subscriber path—should be re-reviewed together with that integration evidence.

[ai-specs]

Additional data point: same leak via a different trigger path - 4xx responses + chunked/streaming requests with split headers/body

We hit the same DelayedExecutionFlowImpl leak on Micronaut 4.10.17 (Kestra fork, webserver mode) - same heap fingerprint (Map:OnErrorResume:FlatMap:OnComplete ~ 3:3:2:1) - but through a different trigger path than the slow keep-alive accumulation described here.

Acute trigger (single request, OOM within minutes):

• HTTP/1.1 Transfer-Encoding: chunked POST
• headers arrive first, body arrives ~300ms later (split arrival)
• the target path returns 404 (either a route-mismatch framework 404, or a controller-returned 404)
• authenticated (unauthenticated requests are intercepted by the auth filter before routing ? 401, no trigger)

Necessary conditions (verified):

• 404/4xx is required - same request shape against a real route returning 202/200 is completely fine (heap flat)
• chunked + split arrival is required - full body sent at once does not trigger
• headers-only with no body ? server waits silently, no response, no spin

Evidence:

• Heap histogram at OOM matches this issue exactly: Map:OnErrorResume:FlatMap:OnComplete ~ 3:3:2:1 (~56M / 56M / 37M / 19M nodes in a single request's worth of minutes)
• MAT: the leaking request holds a StreamingNettyByteBody (body never fully read) and the GC root is IdleStateHandler$ReaderIdleTimeoutTask (connection parked waiting for the rest of the body)
• Chain growth ~ 2.8M nodes/sec - a self-sustaining drain loop within a single request, not "+8 per request"

Suggested addition to the root-cause analysis: after a 404/4xx response is written, Micronaut still needs to drain the request body before the keep-alive connection can be reused. That drain appears to run on a flow path that keeps appending DelayedExecutionFlowImpl steps without ever completing (the #11546 cleanup only runs on normal completion). So the error-path body drain may be a separate accumulation entry point from defuse() - worth checking which flow executes the post-error drain and why it never completes.

Connection: close terminates the leak (verified both ways):

• Client closing the connection mid-body ? heap goes flat immediately (storm stops)
• This matches the workaround in this issue (limit requests per connection + Connection: close)

Suggested framework-level fix direction: on 4xx/5xx with an unconsumed request body, actively close the connection (or consume the body quickly) instead of draining it through the reactive flow path.

Minimal repro (python socket, no framework code needed): authenticated client sends POST <any-404-path> HTTP/1.1 + Transfer-Encoding: chunked headers, waits ~300ms, then sends 2\r\n{}\r\n0\r\n\r\n. Heap explodes within minutes.

Related: this is the same leak as Kestra#17620 (PUT to a non-existent flow ? 404 ? OOM, two production outages). Kestra's fix Kestra#17633 (throw HttpStatusException on error paths so the framework error handler produces the 404, instead of an early raw-404 response) mitigates controller-level 404s - but not route-mismatch 404s - consistent with the drain path being framework-level.