Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Implement getReaderForRange in the RapidsShuffleManager #362

Closed
abellina opened this issue Jul 15, 2020 · 1 comment
Closed

[BUG] Implement getReaderForRange in the RapidsShuffleManager #362

abellina opened this issue Jul 15, 2020 · 1 comment
Labels
bug Something isn't working shuffle things that impact the shuffle plugin

Comments

@abellina
Copy link
Collaborator

If we have a skewed join that has GPU written output, we may end up with needing to read by range.

Currently the RapidsShuffleManager will "delegate" to the cpu shuffle for this, but this will fail (the output was written the catalog, not to files).

This is to implement getReaderForRange

Note that in spark 3.1, getReader became final, and we should only use getReaderForRange.
@abellina abellina added bug Something isn't working ? - Needs Triage Need team to review and classify shuffle things that impact the shuffle plugin labels Jul 15, 2020
@abellina abellina removed the ? - Needs Triage Need team to review and classify label Jul 15, 2020
@abellina abellina mentioned this issue Jul 15, 2020
Merged
@sameerz
Copy link
Collaborator

sameerz commented Aug 19, 2020

Closing in favor of issue #455

@sameerz sameerz closed this as completed Aug 19, 2020
tgravescs pushed a commit to tgravescs/spark-rapids that referenced this issue Nov 30, 2023
@revans2
Add new native parquet footer API and deprecate the old one (NVIDIA#362)
ba5de10 
Signed-off-by: Robert (Bobby) Evans <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working shuffle things that impact the shuffle plugin
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@abellina @sameerz