Ballista shuffle is finally working as intended, providing scalable distributed joins #750
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@houqp @Dandandan @edrevo @alamb @jorgecarleitao Ballista is finally working with scalable distributed joins, at least it is for TPC-H. I plan on following up with some further smaller code cleanup PRs now that the functionality is working. |
alamb
approved these changes
Jul 20, 2021
| @@ -69,7 +78,8 @@ impl ExecutionPlan for UnresolvedShuffleExec { | |||
| } | |||
|
|
|||
| fn output_partitioning(&self) -> Partitioning { | |||
| Partitioning::UnknownPartitioning(self.partition_count) | |||
| //TODO the output partition is known and should be populated here! | |||
There was a problem hiding this comment.
is this something that you want to finish up in this PR?
There was a problem hiding this comment.
I've filed https://github.com/apache/arrow-datafusion/issues/758 as a follow-up for implementing this since it involves more serde work.
I've added an additional test to check that TPC-H query 12 gets planned with correct partitioning information in the shuffle readers. |
|
🎉 |
andygrove
pushed a commit
to andygrove/datafusion
that referenced
this pull request
Jan 31, 2025
* feat: Optimze CreateNamedStruct preserve dictionaries Instead of serializing the return data_type we just serialize the field names. The original implmentation was done as it lead to slightly simpler implementation, but it clear from apache#750 that this was the wrong choice and leads to issues with the physical data_type. * Support dictionary data_types in StructVector and MapVector * Add length checks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Builds on #738 Closes #707.
With this PR we finally have scalable distributed joins.
Query 12 performance at SF=100
Integration tests pass.
Rationale for this change
This is making Ballista work as it was intended to work.
What changes are included in this PR?
Tons of bug fixes around shuffles.
Are there any user-facing changes?
No