Add optional traversal limit for commit offset lookup#4007
Add optional traversal limit for commit offset lookup#4007NssGourav wants to merge 2 commits intoapache:mainfrom
Conversation
Introduce a PersistenceParams knob to bound offset resolution in CommitsImpl and fail fast when exceeded, with coverage verifying the safeguard for both commitLog variants. Made-with: Cursor
|
A few things worth addressing like the limit is still inside CommitsImpl, but @snazy mentioned the TODO was meant for the call sites (REST/service layer), not the implementation itself worth clarifying with him before going further. There's also a counter bug where commitLog and commitLogReversed count traversals differently since traversed starts at 0 in one and 1 in the other, so the same maxTraversal value behaves inconsistently. On the exception side, IllegalStateException feels off for an expected operational limit a dedicated exception would be easier to catch specifically. For the tests, the .hasNext() in the commitLog test is misleading since the exception fires during the commitLog() call itself, not lazily, and there's also no test for the success case where the offset is found within the limit. Overall the approach is reasonable, just needs a quick sync with the maintainer on the architectural question first. |
Use a dedicated exception for limit exceedance, make traversal counting consistent across log variants, and expand tests to cover both fail-fast and success paths. Made-with: Cursor
|
Thanks agreed on the feedback. I understand the TODOs may be intended for call-site guardrails; I implemented an opt-in safeguard in Fixed traversal counting so commitLog and commitLogReversed behave consistently. Updated tests to assert at the right point and added success path coverage. |
Summary
polaris.persistence.commit-offset-lookup-max-traversalparameter (default0= unlimited) to optionally bound commit history traversal while resolving offsets.CommitsImpl.commitLog()andCommitsImpl.commitLogReversed()during offset lookup, failing fast with a clear exception when exceeded.Rationale
CommitsImplcurrently walks commit history unbounded when resolving offsets. In pathological cases (very large histories or missing/far-back offsets) this can lead to unbounded work.The safeguard is opt-in (default unlimited) to avoid breaking background/maintenance flows that may legitimately traverse the full log.
Test plan
./gradlew :persistence:nosql:persistence-impl:testFixes #4000