[FLINK-38464] Introduce OrderedMultiSetState #27071

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

rkhachatryan merged 2 commits into apache:master from rkhachatryan:f38464

Oct 15, 2025

Contributor

rkhachatryan commented Sep 30, 2025 •

edited

Loading

Introduce OrderedMultiSetState and 3 implementations (map, value, adaptive) to be used in SInkUpsertMaterializerV2.

Test coverage is currently provided on the operator level (#27070).

I'm planning to add lower-level unit tests later to this PR.

rkhachatryan mentioned this pull request

[FLINK-38461] Introduce SinkUpsertMaterializerV2 #27070

Open

Collaborator

flinkbot commented Sep 30, 2025 •

edited

Loading

CI report:

431baa7 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

rkhachatryan force-pushed the f38464 branch from 2fb3bb9 to c87cf49 Compare

October 1, 2025 08:23

rkhachatryan marked this pull request as ready for review

October 1, 2025 08:40

rkhachatryan requested a review from pnowojski

October 1, 2025 08:40

davidradl reviewed

View reviewed changes

.../main/java/org/apache/flink/table/runtime/sequencedmultisetstate/SequencedMultiSetState.java Outdated

    
                  /**

                   * Remove the given element. If there are multiple instances of the same element, remove the

                   * first one in insertion order.

Contributor

davidradl Oct 2, 2025

I am curious : should we allow the user to choose LIFO or FIFO for the remove ?

Contributor Author

rkhachatryan Oct 14, 2025

While reviewing the potential usages of this data structure (listed in the FLIP document) I couldn't find any that would require removal of the last element.

davidradl reviewed

View reviewed changes

.../main/java/org/apache/flink/table/runtime/sequencedmultisetstate/SequencedMultiSetState.java Outdated Show resolved Hide resolved

davidradl reviewed

View reviewed changes

.../main/java/org/apache/flink/table/runtime/sequencedmultisetstate/SequencedMultiSetState.java Outdated

    
                       * An element was removed, it was not the most recently added, there are more elements. The

                       * result will not contain any elements

                       */

                      REMOVED_OTHER

Contributor

davidradl Oct 2, 2025

I am curious why nothing is returned in this case, this seems inconsistent with REMOVED_LAST_ADDED which will return the element added before it.

Contributor Author

rkhachatryan Oct 14, 2025

There are no use-cases requiring the element removed "from the middle" (or from the beginning) of the data structure.
I think generalizing the contract here to something like "return a row always except for NOTHING_REMOVED" might make it actually more confusing because the semantics is different in different cases: return the removed element in case of ALL_REMOVED, return the new last element in case of REMOVED_LAST_ADDED.

davidradl reviewed

View reviewed changes

.../main/java/org/apache/flink/table/runtime/sequencedmultisetstate/SequencedMultiSetState.java

    
                  SizeChangeInfo append(T element, long timestamp) throws Exception;

                  /** Get iterator over all remaining elements and their timestamps, in order of insertion. */

                  Iterator<Tuple2<T, Long>> iterator() throws Exception;

Contributor

davidradl Oct 2, 2025 •

edited

Loading

if the multi set changes (i.e. there is a removal) under the iterator what will happen? AI unit test for this would be good. It would be useful to understand any locking that has been considered or is in place.

Contributor Author

rkhachatryan Oct 14, 2025

This depends on the implementation:

in case of LinkedMultiSetState, the iteration is over a copy of the state (so change has no impact)
in case of ValueStateMultiSetState, it uses ArrayList.iterator(), which is fail-fast

But since the client code should not make any assumptions about the implementation, I don't think this should be part of the contract.

pnowojski reviewed

View reviewed changes

Contributor

pnowojski left a comment

(partial review, I haven't yet reviewed linked variant and tests).

.../main/java/org/apache/flink/table/runtime/sequencedmultisetstate/SequencedMultiSetState.java Outdated Show resolved Hide resolved

...n/java/org/apache/flink/table/runtime/orderedmultisetstate/AdaptiveOrderedMultiSetState.java Outdated Show resolved Hide resolved

...n/java/org/apache/flink/table/runtime/orderedmultisetstate/AdaptiveOrderedMultiSetState.java Outdated Show resolved Hide resolved

...main/java/org/apache/flink/table/runtime/sequencedmultisetstate/ValueStateMultiSetState.java Show resolved Hide resolved

...c/main/java/org/apache/flink/table/runtime/orderedmultisetstate/ValueStateMultiSetState.java Outdated Show resolved Hide resolved

github-actions bot added the community-reviewed label

pnowojski requested changes

View reviewed changes

.../src/main/java/org/apache/flink/table/runtime/orderedmultisetstate/OrderedMultiSetState.java Outdated Show resolved Hide resolved

...ain/java/org/apache/flink/table/runtime/orderedmultisetstate/linked/LinkedMultiSetState.java Outdated Show resolved Hide resolved

...ain/java/org/apache/flink/table/runtime/orderedmultisetstate/linked/LinkedMultiSetState.java Outdated Show resolved Hide resolved

...ain/java/org/apache/flink/table/runtime/orderedmultisetstate/linked/LinkedMultiSetState.java Outdated Show resolved Hide resolved

.../main/java/org/apache/flink/table/runtime/sequencedmultisetstate/SequencedMultiSetState.java Outdated Show resolved Hide resolved

...ain/java/org/apache/flink/table/runtime/orderedmultisetstate/linked/LinkedMultiSetState.java Outdated Show resolved Hide resolved

rkhachatryan requested a review from pnowojski

October 8, 2025 12:33

rkhachatryan mentioned this pull request

[FLINK-38460][tests] Improve test coverage for SinkUpsertMaterializer #27068

Merged

pnowojski reviewed

View reviewed changes

...t/java/org/apache/flink/table/runtime/sequencedmultisetstate/SequencedMultiSetStateTest.java Outdated Show resolved Hide resolved

...t/java/org/apache/flink/table/runtime/sequencedmultisetstate/SequencedMultiSetStateTest.java Show resolved Hide resolved

davidradl reviewed

View reviewed changes

...t/java/org/apache/flink/table/runtime/sequencedmultisetstate/SequencedMultiSetStateTest.java Show resolved Hide resolved

rkhachatryan requested a review from pnowojski

October 10, 2025 14:38

davidradl reviewed

View reviewed changes

.../src/main/java/org/apache/flink/table/runtime/sequencedmultisetstate/linked/MetaSqnInfo.java Outdated Show resolved Hide resolved

pnowojski reviewed

View reviewed changes

Contributor

pnowojski left a comment

Thanks for the update, as discussed offline, I've shared with you some pointers to still potentially missing test coverage.

rkhachatryan requested a review from pnowojski

October 14, 2025 11:44

pnowojski approved these changes

View reviewed changes

Contributor

pnowojski left a comment •

edited

Loading

Thanks for the update. LGTM % green build & assuming comments from @davidradl are also resolved

Contributor Author

rkhachatryan commented Oct 14, 2025

Thanks a lot for the reviews!
I'll squash the commits, rebase, and merge the PR unless there are any objections

rkhachatryan force-pushed the f38464 branch from b6df363 to bc1279f Compare

October 14, 2025 12:43

rkhachatryan requested a review from davidradl

October 14, 2025 14:42

rkhachatryan added 2 commits

October 14, 2025 23:43


          [hotfix] Make CompositeSerializer factory methods public

5ff7c67


          [FLINK-38464] Introduce OrderedMultiSetState

431baa7

rkhachatryan force-pushed the f38464 branch from bc1279f to 431baa7 Compare

October 14, 2025 23:43

rkhachatryan merged commit 83db8f7 into apache:master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-reviewed