Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a shim for Databricks 11.3 spark330db [databricks] #7152

Merged
merged 92 commits into from
Dec 13, 2022
Merged

Add a shim for Databricks 11.3 spark330db [databricks] #7152

merged 92 commits into from
Dec 13, 2022

Conversation

gerashegalov
Copy link
Collaborator

@gerashegalov gerashegalov commented Nov 23, 2022 •
edited
Loading

  • introduce non330db directories
  • ShimExtractValue
  • GpuPredicateHelper now extends and shims PredicateHelper
  • Allow passing TEST_PARALLEL to test.sh to be able to run integration tests on a small instance
  • No need to override getSparkShimVersion using the same implementation in every shim

Fixes #6879

Co-authored-by: @amahussein
Co-authored-by: @nartal1

Signed-off-by: Gera Shegalov [email protected]

amahussein and others added 30 commits November 17, 2022 19:14
@amahussein
Add support to DB 11.3 ML LTS in databricks build script
552c33c 
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>
@amahussein
Merge branch 'branch-22.12' into rapids-6979
e456cda
@amahussein
address reviews, fix typos, add doc of how to extend script
cad8294 
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>
@gerashegalov
Merge branch 'rapids-6979' into compileDb113_v2
211672b
@gerashegalov
GpuCsvUtils
31740b1 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
GpuIntervalUtils
c05b083 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
IntervalUtils
cb1cb7a 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
SparkDateTimeShims
86b1b9f 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
AnsiUtil
23a63c5 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
TypeSigUtil
2069f13 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
YearParseUtil 141 errors
6ddf256 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
GpuJsonUtils: 140 errors
f275e5d 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
CharVarcharUtilsShims: 139 errors
8910ec8 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
OrcCastingShims: 138 errors
6e9f922
@gerashegalov
bulk: 81 errros
d50de27 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
OrcShims: 70 errors
0726de3
@gerashegalov
AQEUtils: 67 errors
5d639b6
@gerashegalov
DeltaLakeUtils OffsetWindowFunctionMeta: 55 errors
06fe58a
@gerashegalov
48 errors
9dd5026 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
43 errors
e525e38 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
36 errors
a05c644 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
GpuParquetCrypto.scala: 34 errros
0340846 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
AggregationTagging.scala: 32 errors
57287b8 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
ShimCurrentBatchIterator ParquetCVShims: 30 errors
25f82dc 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
21 errors
dc84117
@gerashegalov
5 errors
f393ce0
@gerashegalov
9 errors
359b9e9 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
9 errors
7a655c6 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
6 errors
bd37b74 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
-s
7b14832
@gerashegalov
10.4 compiles
c403b1a
@gerashegalov
11.3 DeltaInvariantViolationException
ab711fd 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
11.3 done
23ddd59 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
11.3 done
75d9e4c 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
10.4 done
43a2529 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
10.4 done
3c7377f 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
scalastyle
1b43ab4 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
Copy link
Collaborator Author

build

This was referenced Dec 11, 2022
Closed
Closed
Closed
jlowe
jlowe reviewed Dec 12, 2022
View reviewed changes
sql-plugin/pom.xml Outdated Show resolved Hide resolved
sql-plugin/pom.xml Outdated Show resolved Hide resolved
sql-plugin/src/main/314/scala/com/nvidia/spark/rapids/shims/SparkShims.scala Outdated Show resolved Hide resolved
...b/scala/com/databricks/sql/transaction/tahoe/rapids/shims/GpuOptimisticTransactionBase.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/321+-db/scala/com/nvidia/spark/rapids/delta/shims/package-shims.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/330+-nondb/scala/com/nvidia/spark/rapids/shims/Spark330PlusShims.scala Outdated Show resolved Hide resolved
.../main/330db/com/databricks/sql/transaction/tahoe/rapids/shims/GpuOptimisticTransaction.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/post320-treenode/scala/com/nvidia/spark/rapids/shims/TreeNode.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/pre320-treenode/scala/com/nvidia/spark/rapids/shims/TreeNode.scala Outdated Show resolved Hide resolved
tgravescs
tgravescs reviewed Dec 12, 2022
View reviewed changes
Copy link
Collaborator

@tgravescs tgravescs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • .../rapids/shims/Spark320until340Shims.scala → .../rapids/shims/Spark320until340Shims.scala

This move now the class name doesn't match the directory name. Not sure its a big deal at this point but wanted to bring up to discuss...

sql-plugin/pom.xml Outdated Show resolved Hide resolved
.../src/main/311until340-all/scala/com/nvidia/spark/rapids/shims/ParquetTimestampNTZShims.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/314/scala/com/nvidia/spark/rapids/shims/SparkShims.scala Outdated Show resolved Hide resolved
This was referenced Dec 13, 2022
Closed
Closed
Closed
Closed
@gerashegalov
Reviews
3a3ed3b 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
Copy link
Collaborator Author

build

@gerashegalov gerashegalov merged commit a42e328 into NVIDIA:branch-23.02 Dec 13, 2022
@gerashegalov gerashegalov deleted the compileDb113_v2 branch December 13, 2022 18:00
@gerashegalov gerashegalov mentioned this pull request Jan 10, 2023
Closed
9 tasks
@rwlee rwlee mentioned this pull request Feb 3, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tgravescs tgravescs tgravescs approved these changes

@jlowe jlowe jlowe approved these changes

@revans2 revans2 Awaiting requested review from revans2 revans2 is a code owner

@GaryShen2008 GaryShen2008 Awaiting requested review from GaryShen2008 GaryShen2008 is a code owner

@NvTimLiu NvTimLiu Awaiting requested review from NvTimLiu NvTimLiu is a code owner

@zhanga5 zhanga5 Awaiting requested review from zhanga5

Assignees

@gerashegalov gerashegalov

Labels
feature request New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEA] Support Databricks 11.3 ML LTS
7 participants
@gerashegalov @revans2 @jlowe @tgravescs @sameerz @amahussein @nartal1