Skip to content

Conversation

tdcmeehan
Copy link
Contributor

No description provided.

@prestodb-ci prestodb-ci added the from:IBM PRs from IBM label Aug 29, 2025
@prestodb-ci prestodb-ci requested review from a team, ShahimSharafudeen and nishithakbhaskaran and removed request for a team August 29, 2025 20:36
@tdcmeehan tdcmeehan force-pushed the conn branch 9 times, most recently from eced569 to 826a238 Compare September 4, 2025 16:11
Copy link
Contributor

@BryanCutler BryanCutler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @tdcmeehan. I mostly just had some questions about auth with the flight server

@tdcmeehan tdcmeehan force-pushed the conn branch 3 times, most recently from bdfb6b2 to 79aed22 Compare September 9, 2025 15:12
Copy link

@aditi-pandit aditi-pandit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RFC looks good overall. Have only a minor comment.

@tdcmeehan tdcmeehan force-pushed the conn branch 2 times, most recently from a4ddced to 7196f6d Compare September 16, 2025 19:40
Copy link
Contributor

@pdabre12 pdabre12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @tdcmeehan , LTGM.

Should we add some documentation on how to load up the new ArrowFederationConnector?

Copy link
Contributor

@BryanCutler BryanCutler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just a minor clarification to the split scheduling section


#### Split Scheduling Phase
1. Coordinator's single-node connector (e.g., JDBC, MongoDB) creates its connector-specific split
2. Split is scheduled to C++ worker as `ArrowFederationSplit` containing serialized split bytes
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to clarify, the coordinator will send a connector-specific split to the C++ worker, but it will be handled by the ArrowFederation protocol where it will be read as bytes only.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, made that a bit more clear.

@czentgr
Copy link

czentgr commented Sep 18, 2025

FYI RFC-0013 exists already. Next one would be RFC-0016.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

from:IBM PRs from IBM

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants