Add VirtualObjectStore to support routing paths to multiple ObjectStores #17084
+390
−2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…and deterministic sorting
…s and collect remaining path segments.
…oining base and meta locations
github-actions
bot
added
execution
…icate that write operations are not supported
kosiew
force-pushed
the
virtual-object-store-16991
branch
from
0f3e862 to
df7fa25
Compare
|
FYI @EmilyMatt -- I am not sure if you have seen what @kosiew is working on |
- Add chrono to [dependencies] in datafusion/execution/Cargo.toml - Remove chrono from [dev-dependencies] - Update tokio dev-dependency to enable features ["macros", "rt", "sync"]
kosiew
force-pushed
the
virtual-object-store-16991
branch
from
789630a to
5075467
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #16991.
Rationale for this change
DataFusion currently lacks a mechanism to unify access to multiple object stores under a single abstraction. This PR introduces a
VirtualObjectStorethat routes read operations to specific object stores based on the first segment of the object path (i.e., a virtual prefix). This enables more flexible configurations where datasets might be spread across S3, local disk, memory, etc., but can be accessed transparently.What changes are included in this PR?
VirtualObjectStoreimplementation that supports routing of read operations (e.g.,get,list,list_with_delimiter) to configured stores based on path prefix.virtual_store: Option<Arc<dyn ObjectStore>>intoFileScanConfigandFileScanConfigBuilder, enabling its use in scans.DataSource::executeto use the virtual store if configured, falling back to resolving the store viaobject_store_url.async-trait,tokio) for async trait implementation and testing.Are these changes tested?
Yes. The PR includes extensive unit tests under
virtual_object_store.rscovering:listandlist_with_delimitersorting and resultsAre there any user-facing changes?
Yes:
virtual_storetoFileScanConfig, enabling custom routing behavior.No breaking changes were introduced. Write operations (
put,delete, etc.) are not yet supported by theVirtualObjectStoreand returnNotSupportederrors accordingly.