fix: Scheduler panic routing errors #4097
Conversation
|
Until the fix for #4100 is merged, clippy will likely fail on this PR as well |
| // The query output partition number may not equal current pipeline's output partition number, | ||
| // but the query output has at least one partition， | ||
| // so send error to the first partition of the query output. |
There was a problem hiding this comment.
| // The query output partition number may not equal current pipeline's output partition number, | |
| // but the query output has at least one partition， | |
| // so send error to the first partition of the query output. | |
| // The query output partitioning may not match the current pipeline's | |
| // but the query output has at least one partition | |
| // so send error to the first partition of the query output. |
datafusion/core/src/scheduler/mod.rs
Outdated
| let scheduled: Vec<_> = stream.try_collect().await.unwrap_or(vec![]); | ||
| let expected = query.collect().await.unwrap_or(vec![]); |
There was a problem hiding this comment.
| let scheduled: Vec<_> = stream.try_collect().await.unwrap_or(vec![]); | |
| let expected = query.collect().await.unwrap_or(vec![]); | |
| let scheduled: Vec<_> = stream.try_collect().await.unwrap_or_default(); | |
| let expected = query.collect().await.unwrap_or_default(); |
tustvold
changed the title
|
I took the liberty of fixing the clippy lints and making a minor change to the docs so that we can get this in quickly for you, thanks again 👍 |
Thank you, sorry, I didn't see your reply, I just resubmitted @tustvold |
|
No worries at all 👍 FWIW it can help reviewers if you avoid rebasing reviewed PRs as reviewers can then easily see what has changed, everything then gets squashed on merge. It makes no difference on a PR this size, but thought I would mention it for future contributions 😄 |
Thanks for your pointing👍 |
|
Benchmark runs are scheduled for baseline = f61b43a and contender = 7e944ed. 7e944ed is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
Which issue does this PR close?
Closes #4096 .
Rationale for this change
The query output partition number may not equal current pipeline's output partition number, so send error to the partition of the query output will cause panic.
What changes are included in this PR?
Because the query output has at least one partition，so we send error that thrown by non-output pipelines to the first partition of the query output.
Are there any user-facing changes?