Skip to content

Fix memory leak and re-enable immutable-send GC optimization#4944

Open
SeanTAllen wants to merge 4 commits intomainfrom
fix-selector-painting-4102
Open

Fix memory leak and re-enable immutable-send GC optimization#4944
SeanTAllen wants to merge 4 commits intomainfrom
fix-selector-painting-4102

Conversation

@SeanTAllen
Copy link
Member

@SeanTAllen SeanTAllen commented Mar 7, 2026

When a behavior is called through a trait and the concrete actor's parameter has a different trace-significant capability (e.g., trait has iso, concrete has val), the sender traces with one trace kind and the receiver traces with another. This ORCA GC mismatch causes field reference counts to never reach zero, leaking objects reachable from the parameter.

The fix includes trace-kind characters in mangled names for behaviors and constructors, so methods whose parameters differ in trace-significant ways get distinct vtable indices. When a forwarding method is created due to a cap mismatch, genfun_forward adds a dispatch case that traces with the forwarding method's params (the trait's capabilities) but calls the concrete method's handler. A two-pass ordering in genfun_method_bodies ensures concrete handlers are generated before forwarding dispatch cases that reference them.

With the leak fixed, the might_reference_actor optimization can safely be re-enabled. The compiler now computes at compile time whether each type could transitively contain an actor reference using a monotone fixpoint, and emits the result in the type descriptor instead of the hardcoded true that was set as a safety net in PR #4256. The runtime code in gc.c is already correct — it just needed a truthful value.

This is a multi-type PR (Fixed + Changed). CHANGELOG and release notes are updated directly; no changelog label should be applied.

Design: #4943

Closes #4102

@SeanTAllen SeanTAllen added the changelog - fixed Automatically add "Fixed" CHANGELOG entry on merge label Mar 7, 2026
@ponylang-main ponylang-main added the discuss during sync Should be discussed during an upcoming sync label Mar 7, 2026
@SeanTAllen SeanTAllen changed the title Fix memory leak when a behavior parameter has a different capability than the trait it implements Fix memory leak Mar 7, 2026
@SeanTAllen SeanTAllen force-pushed the fix-selector-painting-4102 branch 2 times, most recently from 4f5c8ec to daab566 Compare March 7, 2026 19:53
@SeanTAllen SeanTAllen added do not merge This PR should not be merged at this time and removed changelog - fixed Automatically add "Fixed" CHANGELOG entry on merge labels Mar 7, 2026
@SeanTAllen SeanTAllen changed the title Fix memory leak Fix memory leak and re-enable immutable-send GC optimization Mar 7, 2026
@SeanTAllen SeanTAllen removed the do not merge This PR should not be merged at this time label Mar 7, 2026
@SeanTAllen
Copy link
Member Author

I tried to do these individually but there are so intertwined that I decided they needed to come in together.

@SeanTAllen SeanTAllen force-pushed the fix-selector-painting-4102 branch 3 times, most recently from 36dbcd0 to 358471e Compare March 8, 2026 03:09
…than the trait it implements

When a behavior is called through a trait and the concrete actor's parameter
has a different trace-significant capability (e.g., trait has iso, concrete
has val), the sender traces with one trace kind and the receiver traces with
another. This ORCA GC mismatch causes field reference counts to never reach
zero, leaking objects reachable from the parameter.

The fix includes trace-kind characters in mangled names for behaviors and
constructors, so methods whose parameters differ in trace-significant ways
get distinct vtable indices. When a forwarding method is created due to a
cap mismatch, genfun_forward now adds a dispatch case that traces with the
forwarding method's params (the trait's capabilities) but calls the concrete
method's handler. A two-pass ordering in genfun_method_bodies ensures
concrete handlers are generated before forwarding dispatch cases that
reference them.

The leak is not currently active because make_might_reference_actor forces
full tracing of immutable objects. Without this fix, re-enabling that
optimization would expose the leak.

Design: #4943
Closes #4102
…actor

make_might_reference_actor currently always returns true, so immutable
tracing recurses into fields. The test incorrectly assumed it would skip
field recursion for non-actor fields.
The might_reference_actor optimization was disabled in PR #4256 as a
safety net — hardcoded to true for all types, forcing full tracing of
every immutable object. This caused a ~1/3 performance drop in
message-heavy programs.

The compiler now computes might_reference_actor at compile time using a
monotone fixpoint: actors are always true, classes/structs/tuples
propagate from their fields (including through trait subtypes), and
primitives are always false. The result is baked into the type descriptor
so the runtime reads a constant — no runtime computation needed.

The existing runtime code in gc.c is already correct — it checks
might_reference_actor at all 11 call sites. Only the compiler needed to
change: emit the computed value instead of hardcoded true.

Adds a new full-program test (val-to-val-with-actor-through-trait) that
verifies immutable tracing does recurse when B has an actor field.
Counterfactual-verified: hardcoded true breaks the val-to-val test,
hardcoded false causes regression-1118 to segfault.

Release notes and CHANGELOG updated directly (multi-type PR: Fixed +
Changed). Individual release notes file removed per convention.
@SeanTAllen SeanTAllen force-pushed the fix-selector-painting-4102 branch from 358471e to bcedd41 Compare March 8, 2026 03:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

discuss during sync Should be discussed during an upcoming sync

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Memory leak when val parameter implements iso parameter trait.

2 participants