-
Notifications
You must be signed in to change notification settings - Fork 403
Description
Search before asking
- I searched in the issues and found nothing similar.
Motivation
Currently, Fluss primary-key tables allow delete operations by default. However, certain use cases, such as delta joins or log deduplication (expected append-only changelogs) may want to prevent deletions.
While Fluss provides a sink connector option, sink.ignore.delete
, which silently drops delete records at the sink, this approach is not sufficient for system-level guarantees. Connector options can be overridden dynamically via SQL hints, making them unsuitable for enforcing strict table-level semantics.
To ensure strong, immutable guarantees that no deletions occur in the changelog stream, we propose a table-level configuration (e.g., table.delete.behavior
) that is enforced by the system itself, either by rejecting deletes with an error or silently ignoring them, regardless of connector settings or client-side overrides.
This would allow users to safely declare a table as append-only, upsert-only enabling reliable downstream processing for use cases like delta joins.
Solution
We propose to introduce a unified, enum-based configuration to explicitly control this behavior on server side:
table.delete.behavior = [allow | disable | ignore]
allow
(default): Allow normal delete operations.ignore
: Silently skip delete requests without error (may expose a metric for this case in the future).disable
: Reject delete requests with a clear error message (e.g., for strict immutability).
We should also update org.apache.fluss.flink.source.FlinkTableSource#getChangelogMode
to reflect this behavior that no deletions in the generated changelogs.
Anything else?
We should add checks when creating tables with this option set:
- Only pk tables can set this option explicitly, log tables with this option should throw exception.
- Currently, the
FIRST_ROW
andVERSIONED
merge engines do not support delete operations.
Therefore, tables using these engines should automatically default totable.delete.behavior=ignore
.
Users may explicitly settable.delete.behavior=disable
to reject deletions,
but settingtable.delete.behavior=allow
must throw an exception, as deletes are fundamentally unsupported by these merge engines.
Willingness to contribute
- I'm willing to submit a PR!