Reviving #14411 - introducing on-demand repartitioning #17020
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
documentation
|
Thanks @AdamGS -- can you remind me what type of queries this should benefit? I am happy to start running some benchmarks / start reviewing some code cc @crepererum |
|
My understanding is that it should be beneficial in queries where the different partitions of the physical plan have heavy skew, allowing "lighter" partitions to pick up more work, which should both reduce the effect of these long-tail anomalies and reduce memory usage in cases where some producers are faster then their upstream consumer. I'll work to fix the remaining test issues, seems like the plan changed a bit since the original code was written. |
AdamGS
force-pushed
the
on-demand-repartition-with-config
branch
from
574c138 to
9112ac9
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR closes #14287, but is actually just a squash + rebase of #14411 by @Weijun-H., see the original PR for all the benchmarking results, rationale and thinking.
AFAICT there are no open discussions/comments there, but I would be happy to at least try and get this PR over the line as it seems like it offers some nice performance benefits.