-
Notifications
You must be signed in to change notification settings - Fork 32
Add RFC-0013 for Java connector federation in C++ workers #46
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
eced569
to
826a238
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks @tdcmeehan. I mostly just had some questions about auth with the flight server
bdfb6b2
to
79aed22
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
RFC looks good overall. Have only a minor comment.
a4ddced
to
7196f6d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @tdcmeehan , LTGM.
Should we add some documentation on how to load up the new ArrowFederationConnector
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, just a minor clarification to the split scheduling section
RFC-0013-java-connector-federaton.md
Outdated
|
||
#### Split Scheduling Phase | ||
1. Coordinator's single-node connector (e.g., JDBC, MongoDB) creates its connector-specific split | ||
2. Split is scheduled to C++ worker as `ArrowFederationSplit` containing serialized split bytes |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just to clarify, the coordinator will send a connector-specific split to the C++ worker, but it will be handled by the ArrowFederation
protocol where it will be read as bytes only.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, made that a bit more clear.
FYI |
No description provided.