branch-3.1: [fix](compaction) Avoid repeatedly compacting large cumulative rowset #64954 #65470 - #67600
Open
Yukang-Lian wants to merge 2 commits into
Open
Conversation
…apache#64954) - Remove the promotion-size early return in size-based cumulative compaction rowset selection so level-size trimming can exclude an oversized leading rowset. - Keep the level-size trimming loop from trimming a single candidate rowset to an empty input set. - Apply the same behavior to both cloud and storage cumulative compaction policies. - Add regression coverage for the large-head/small-tail case and the single-rowset guard in both paths. (cherry picked from commit a1e076e)
…ng (apache#65470) Issue Number: None Related PR: apache#59268 Problem Summary: When the cumulative compaction max score is reduced, back trimming can leave a single non-overlapping rowset. It cannot be compacted alone, so compaction repeatedly returns `CUMULATIVE_NO_SUITABLE_VERSION` without advancing the cumulative point. This change remembers the last trimmed rowset in both cloud and local size-based policies. If trimming strands a single non-overlapping rowset, it restores the direct successor and allows one max-score overshoot so the input remains mergeable. Existing overlapping-singleton and cloud empty-rowset behavior is unchanged. Prevent max-score trimming from leaving an unmergeable cumulative compaction input. (cherry picked from commit 54300f9)
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Collaborator
Author
|
run buildall |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #64954, #65470
Problem Summary: Backport the cumulative-compaction large-head fix from #64954 to
branch-3.1, together with the final mergeable-input protection from #65470.#64954 removes the promotion-size early return so level-size trimming can exclude an oversized leading rowset instead of repeatedly compacting it. It also keeps the last suffix rowset available for the singleton checks. The same behavior is applied to the cloud and local size-based policies.
branch-3.1predates the deferred max-score tail-trimming mechanism changed by #65470, so that later implementation cannot be cherry-picked directly. This backport preserves #65470's final invariant in the branch's collection-time guard: trimming must not strand a single non-overlapping rowset when its direct successor is needed to form a mergeable input. Focused cloud and local regression cases cover both the stranded-head and overlapping-singleton max-score scenarios.The source commits are
a1e076e8d6969feafd8411f7b19e22664f43c543(#64954) and54300f922fddbeac6e679a83d7829444d4a72f77(#65470). The branch is based on the latestbranch-3.1at publication time.Release note
Avoid repeatedly compacting an oversized cumulative rowset while keeping trimmed inputs mergeable on branch-3.1.
Check List (For Author)
git diff --check: passed