Conversation momentum implementation - #3314
Open
QuMuzafferEge wants to merge 6 commits into
Open
Conversation
Member
|
Please give this PR a descriptive title before merging. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in
routing.momentumpolicy block to the router engine to fix route flapping in multi-turn conversations. Todaymin_chars/max_charsonly ever measure the latest user turn, so a long coding question correctly escalates to a strong model viamin_chars, but the very next short follow-up ("yes, do it") falls straight back todefault_modelmid-task — paying a real switch cost (model load/eviction, KV re-prefill, provider cache miss) every time.routing.momentumreplaces the raw last-turn byte length with a momentum-filtered "effective length," recomputed fresh from each request's own message history (no server-side session state — the conversation is already resent every turn, same as any OpenAI-style chat client):json
"routing": {
"momentum": { "enabled": true, "attack": 1.0, "release": 0.3 }
}
An asymmetric EMA over each user turn's UTF-8 byte length: attack (default 1.0) weights a rising turn so a sudden long/complex turn escalates immediately, release (default 0.3) weights a falling turn so a short follow-up only decays the effective length by 30% per turn — keeping the conversation on the escalated route for several turns before de-escalating. attack = release = 1.0 degenerates exactly to today's behavior. Applies only to min_chars/max_chars; absent or enabled: false is behavior-identical to a policy without the block.
Fixes #2956
Scope
This PR addresses one clear issue or change.
I reviewed the full diff myself before submitting.
I removed unrelated local changes.
Testing
Code builds without errors locally.
I tested this change locally.
I described the testing performed below.
Testing details:
Built and tested on Linux (Ubuntu 24.04 container, matching the cpp-unit-tests CI job's recipe: cmake --preset default → cmake --build --preset default --target cpp-ci-tests → ctest -L "^cpp-ci$").
50/52 cpp-ci tests passed, including every test this change touches:
RoutingPolicyMomentumTest (new) — filter arithmetic (rise/fall/tie/degenerate coefficients, IEEE-exactness), CharsCondition/EvalContext::effective_chars wiring, engine-level fold, trace rationale format.
RoutingPolicyParserTest (extended) — momentum block validation (required enabled, (0,1] coefficient bounds, unknown-key rejection) and schema/parser key parity.
RoutingClassifierServicesTest (extended) — build_route_context collecting user_turn_chars across chat messages, legacy prompt, and Responses input forms.
RoutingConformanceCorpusTest — two new golden groups (l1_momentum_effective_chars, l1_momentum_absent_is_noop) plus every pre-existing group re-validated byte-for-byte unchanged, confirming no regression when the block is absent.
2 pre-existing failures (TelemetryHelpersTest, CliRuntimeOverrideTest) are unrelated to this change — neither file is touched by this diff, and both look like environment artifacts of a from-scratch container vs. the pinned CI image (locale/encoding and a streaming-timing sensitivity respectively).
python test/test_schema_lock.py and python -m unittest test.test_routing_fixtures both pass (schema-lock hash refreshed for the additive routing.momentum schema keys and the decision.schema.json rationale-field description update).
Documentation
Documentation is affected and has been updated.
Added a "Momentum (routing.momentum)" section to docs/dev/router-policy.md (field table, formula, worked example, trace format, explicit non-goals), footnoted the min_chars/max_chars schema descriptions and the frozen-semantics table in src/cpp/resources/schemas/README.md to reference the new opt-in mechanism without redefining the existing frozen unit/comparator.
Breaking Changes
This PR does not introduce breaking changes.
Fully additive and opt-in: new RoutePolicy/RouteContext/EvalContext fields (C++-internal), one new optional routing.momentum schema property, and use of the already-optional trace_entry.rationale field. No existing field is redefined, and every code path is gated behind momentum.enabled (default false), so any policy without the block is behavior-identical to before this change.