Skip to content

[api][core] Reset Format Table partition locations on overwrite - #9710

Open
sundapeng wants to merge 2 commits into
apache:masterfrom
sundapeng:feature/format-table-partition-location-reset-contract
Open

[api][core] Reset Format Table partition locations on overwrite#9710
sundapeng wants to merge 2 commits into
apache:masterfrom
sundapeng:feature/format-table-partition-location-reset-contract

Conversation

@sundapeng

@sundapeng sundapeng commented Sep 9, 2026

Copy link
Copy Markdown
Member

Purpose

Follow-up to #9540. Every Format Table write commit currently reads the whole partition registry to find out whether it touches a partition registered at a custom location, and it rejects overwrite and truncate of such partitions. On a table with hundreds of thousands of partitions that is hundreds of REST calls per commit, for every writer.

This PR changes both sides:

  • REST contract: partitionOptions[i].path = null resets a partition to its default location. It is accepted only with replaceStatistics=true and a partitionStatistics entry for the same spec, both applied in the same request. Omitting path keeps the stored location; non-null paths, other options and statistics-only requests behave as before. Additive statistics for a partition that already has a custom location are rejected. Servers without the reset reject the request, since a null option value already fails their validation.
  • FormatTableCommit: appends look up only the partitions they wrote, static overwrites list only their prefix, and overwrite and truncate reset the target partitions to the default location together with replacement statistics instead of failing. External data is left untouched. Whole-table overwrite and truncate still list the table, which they need for directory cleanup.

Tests

  • RESTApiJsonTest, MockRESTCatalogTest, CatalogFormatTablePartitionManagerTest
  • FormatTableCommitTest, FormatTableCommitStatisticsTest, FormatTableCommitRegistryValidationTest
  • node scripts/validate-rest-openapi.js

@sundapeng sundapeng changed the title [api][core] Add REST contract for resetting partition locations [api][core] Define REST contract for partition location reset Sep 9, 2026
@sundapeng
sundapeng marked this pull request as draft September 9, 2026 22:50
@sundapeng
sundapeng force-pushed the feature/format-table-partition-location-reset-contract branch from 5aeae8b to 606f96d Compare September 9, 2026 22:56
A null path in partitionOptions resets an existing Format Table partition to
its default location, together with replacement statistics for that
partition; non-null paths and statistics-only requests keep their behavior.
@sundapeng
sundapeng force-pushed the feature/format-table-partition-location-reset-contract branch from 606f96d to aa5cadf Compare September 9, 2026 23:13
@sundapeng
sundapeng marked this pull request as ready for review September 9, 2026 23:21
@sundapeng sundapeng changed the title [api][core] Define REST contract for partition location reset [api][core] Reset partition locations on overwrite and scope Format Table registry lookups Sep 9, 2026
Resolve append, overwrite, and truncate targets through the narrowest registry API. Reset partition path metadata with replacement statistics while preserving external data.
@sundapeng sundapeng changed the title [api][core] Reset partition locations on overwrite and scope Format Table registry lookups [api][core] Reset Format Table partition locations on overwrite Sep 10, 2026
@JingsongLi

Copy link
Copy Markdown
Contributor

Could we pass the concrete target partition path instead of using path: null as a reset operation?

The writer already knows its output directory. The REST server could derive and normalize the default partition path from the table metadata and partition spec, then remove the stored path override when the requested path equals that default. Whether the server stores this as NULL should be an implementation detail. The API semantics would be:

  • Omit path: keep the current location.
  • Provide path: specify the desired location.

For this PR, we could limit changes to existing locations to switching back to the default directory. Such a change should still require replacement statistics for the same partition and apply both atomically; an ordinary createPartitions(ignoreIfExists=true) should not silently relocate an existing partition.

This would also require recognizing the default path before the custom-location overlap/conflict checks. Passing a non-null path alone does not provide compatibility with older servers, so the updated server contract would still need to be explicit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants