KAFKA-20489: Add enable.transactional.statestores#22141
Open
nicktelford wants to merge 1 commit intoapache:trunkfrom
Open
KAFKA-20489: Add enable.transactional.statestores#22141nicktelford wants to merge 1 commit intoapache:trunkfrom
nicktelford wants to merge 1 commit intoapache:trunkfrom
Conversation
Introduces the `enable.transactional.statestores` config (default false). When enabled, uncommitted writes are held in an in-memory buffer per store and are not flushed to the underlying base store until the Kafka transaction commits, making staged writes invisible to IQ reads at the committed isolation level until the containing commit completes. The config is threaded through TopologyConfig, TaskConfig, task creators, and ProcessorStateManager. When transactional stores are active, the EOS state wipe on unclean shutdown is suppressed: since uncommitted data never reaches the base store, there is nothing to wipe; corruption is handled explicitly via markChangelogAsCorrupted. StreamTask.postCommit is extended to flush the pending write buffer on every commit interval, not only on task revocation or close. Under EOS the normal commit-interval path previously skipped maybeCheckpoint, the only path that calls stateMgr.commit() to flush each store's buffer. Without this, READ_COMMITTED IQ readers see no new data mid-run, and the uncommitted buffer grows unbounded between task lifecycle events.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduces the
enable.transactional.statestoresconfig (default false). When enabled, uncommitted writes are held in an in-memory buffer per store and are not flushed to the underlying base store until the Kafka transaction commits, making staged writes invisible to IQ reads at the committed isolation level until the containing commit completes.The config is threaded through TopologyConfig, TaskConfig, task creators, and ProcessorStateManager. When transactional stores are active, the EOS state wipe on unclean shutdown is suppressed: since uncommitted data never reaches the base store, there is nothing to wipe; corruption is handled explicitly via markChangelogAsCorrupted.
StreamTask.postCommit is extended to flush the pending write buffer on every commit interval, not only on task revocation or close. Under EOS the normal commit-interval path previously skipped maybeCheckpoint, the only path that calls stateMgr.commit() to flush each store's buffer. Without this, READ_COMMITTED IQ readers see no new data mid-run, and the uncommitted buffer grows unbounded between task lifecycle events.
Co-Authored-By: Claude Sonnet 4.6 noreply@anthropic.com