Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] Add shims for 3.2.1-SNAPSHOT #3470

Closed
andygrove opened this issue Sep 13, 2021 · 0 comments · Fixed by #3973
Closed

[FEA] Add shims for 3.2.1-SNAPSHOT #3470

andygrove opened this issue Sep 13, 2021 · 0 comments · Fixed by #3973
Assignees
@gerashegalov
Labels
feature request New feature or request P0 Must have for release
Milestone
Nov 1 - Nov 12

Comments

@andygrove
Copy link
Contributor

Is your feature request related to a problem? Please describe.
Apache Spark is now publishing nightly 3.2.1-SNAPSHOT builds from branch-3.2 and we need to add shims for this version.

Describe the solution you'd like
Add new shims.

Describe alternatives you've considered
None

Additional context
None
@andygrove andygrove added feature request New feature or request ? - Needs Triage Need team to review and classify labels Sep 13, 2021
@Salonijain27 Salonijain27 added P1 Nice to have for release and removed ? - Needs Triage Need team to review and classify labels Sep 14, 2021
@gerashegalov gerashegalov self-assigned this Oct 19, 2021
@gerashegalov gerashegalov added P0 Must have for release and removed P1 Nice to have for release labels Oct 21, 2021
@gerashegalov gerashegalov added this to the Oct 18 - Oct 29 milestone Oct 21, 2021
@gerashegalov gerashegalov mentioned this issue Oct 26, 2021
Merged
gerashegalov added a commit that referenced this issue Oct 29, 2021
@gerashegalov
Prepare for 3.2.1 shim, various shim build fixes and improvements [da…
c6b2479 
…tabricks] (#3923)

Signed-off-by: Gera Shegalov <[email protected]>

Contributes to #3470
Closes #3915  

- Remove unused dependency-reduced-pom-generation from agg pom
- Eliminate double invocation of the shade plugin taking extra 6 seconds by delegating artifact attachment to the build helper plugin
- Improve clean by customizing maven-clean-plugin
- Enable *WithDatabricks profiles in `buildall`
- Generate SparkShimServiceProvider discovery files on the fly
- Refactor dist/pom.xml to reduce copy/paste code by  50%, delegate to default plugins when possible by customization
- Disable doc  generation automatically if the profile does not contain 301 to avoid inadvertent diffs on CI
- Fix typo in verify_same_sha_for_unshimmed
- Remove redundant SparkShims.getRapidsShuffleManagerClass
- Consolidate most of Spark320 shim code into Spark32xShims.
@gerashegalov gerashegalov mentioned this issue Oct 30, 2021
Merged
gerashegalov added a commit that referenced this issue Nov 1, 2021
@gerashegalov
Add shims for Apache Spark 3.2.1-SNAPSHOT builds (#3973)
252d715 
Closes #3470. Adds support to run RAPIDS Accelerator on top of Spark 3.2.1-SNAPSHOT

Signed-off-by: Gera Shegalov <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gerashegalov gerashegalov

Labels
feature request New feature or request P0 Must have for release
Projects
None yet
Milestone
Nov 1 - Nov 12
Development

Successfully merging a pull request may close this issue.

Add shims for Apache Spark 3.2.1-SNAPSHOT builds gerashegalov/spark-rapids
3 participants
@andygrove @gerashegalov @Salonijain27