[Spark 4.0] Account for PartitionedFileUtil.splitFiles signature change.
#10857
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes NVIDIA#10299. In Apache Spark 4.0, the signature of `PartitionedFileUtil.splitFiles` was changed to remove unused parameters (apache/spark@eabea643c74). This causes the Spark RAPIDS plugin build to break with Spark 4.0. This commit introduces a shim to account for the signature change. Signed-off-by: MithunR <[email protected]>
|
Build |
razajafri
reviewed
May 22, 2024
...n/spark341db/scala/org/apache/spark/sql/execution/rapids/shims/PartitionedFileUtilShim.scala
Outdated
Show resolved
Hide resolved
Signed-off-by: MithunR <[email protected]>
|
Build |
mythrocks
changed the title
PartitionedFileUtil.splitFiles signature change.PartitionedFileUtil.splitFiles signature change.
razajafri
reviewed
May 23, 2024
...lugin/src/main/spark341db/scala/com/nvidia/spark/rapids/shims/PartitionedFileUtilsShim.scala
Show resolved
Hide resolved
razajafri
reviewed
May 23, 2024
|
Edit: Fixed. |
|
Build |
razajafri
reviewed
May 28, 2024
sql-plugin/src/main/spark350/scala/com/nvidia/spark/rapids/shims/PartitionedFileUtilsShim.scala
Outdated
Show resolved
Hide resolved
|
I've merged up to pull in #10933. |
razajafri
previously approved these changes
May 29, 2024
|
build |
razajafri
added
Spark 4.0+
|
CI failed with |
|
build |
|
Ah, this PR is also failing CI on the following tangential problem: #10945 should help. |
|
build |
|
Looks like this will need special handling for Databricks: I'll get on this shortly. |
Barked up the wrong tree for a bit. This was only a missing import. Testing the fix now. |
|
Build |
NVnavkumar
approved these changes
May 31, 2024
|
This change has been merged. Thank you for the reviews, @razajafri, @NVnavkumar. |
SurajAralihalli
pushed a commit
to SurajAralihalli/spark-rapids
that referenced
this pull request
Jul 12, 2024
…ange. (NVIDIA#10857) * Account for PartitionedFileUtil.splitFiles signature change. Fixes NVIDIA#10299. In Apache Spark 4.0, the signature of `PartitionedFileUtil.splitFiles` was changed to remove unused parameters (apache/spark@eabea643c74). This causes the Spark RAPIDS plugin build to break with Spark 4.0. This commit introduces a shim to account for the signature change. Signed-off-by: MithunR <[email protected]> * Common base for PartitionFileUtilsShims. Signed-off-by: MithunR <[email protected]> * Reusing existing PartitionedFileUtilsShims. * More refactor, for pre-3.5 compile. * Updated Copyright date. * Fixed style error. * Re-fixed the copyright year. * Added missing import. --------- Signed-off-by: MithunR <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #10299.
In Apache Spark 4.0, the signature of
PartitionedFileUtil.splitFileswas changed to remove unused parameters (apache/spark@eabea643c74). This causes the Spark RAPIDS plugin build to break with Spark 4.0.This commit introduces a shim to account for the signature change.
fixes #10299