Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Spark 2.x Explain Api #4529

Merged
merged 96 commits into from
Jan 20, 2022
Merged
Changes from 1 commit
Commits
Show all changes
96 commits
Select commit Hold shift + click to select a range
8eb81f6
Add an explain only mode configuration to run plugin on CPU to get wh…
tgravescs Dec 1, 2021
1fca5ef
Updates
tgravescs Dec 1, 2021
f043f86
Don't allocate gpu or enable shuffle for explain only mode
tgravescs Dec 2, 2021
431426c
explain only mode check for rapids shuffle internal manager
tgravescs Dec 2, 2021
3541832
update doc
tgravescs Dec 2, 2021
6ec51d0
Change how we check explain with sql enabled
tgravescs Dec 3, 2021
21a79ca
Merge branch 'explainonlymode' of github.com:tgravescs/spark-rapids i…
tgravescs Dec 3, 2021
b8e9e26
update not work message:
tgravescs Dec 3, 2021
c15b786
Merge remote-tracking branch 'origin/branch-22.02' into explainonlymode
tgravescs Dec 3, 2021
ba3b355
fix spacing
tgravescs Dec 6, 2021
42aa80f
Update doc adding explain option
tgravescs Dec 6, 2021
278d96c
update docs
tgravescs Dec 6, 2021
2a3b228
Add explian only mode test to make sure runs on cpu
tgravescs Dec 6, 2021
3bdf4f2
add note about adaptive
tgravescs Dec 7, 2021
4b88718
get rest of broadcast shims
tgravescs Dec 7, 2021
8f9301a
Update logging of enabled and explain only mode
tgravescs Dec 7, 2021
d79c9a4
update config docs
tgravescs Dec 7, 2021
ca880f1
Add spark2 module that supports the Explain only api
tgravescs Jan 10, 2022
c961f12
copyrights
tgravescs Jan 10, 2022
1e5b1ab
imports
tgravescs Jan 10, 2022
b0f815c
Revert "imports"
tgravescs Jan 10, 2022
25b1a0a
imports
tgravescs Jan 10, 2022
7f5e4bb
cleanup imports
tgravescs Jan 10, 2022
5cc57dc
cleanup imports datasource
tgravescs Jan 10, 2022
d981107
remove cudf as dependency
tgravescs Jan 10, 2022
645c39b
upmerge joins to latest
tgravescs Jan 10, 2022
879c005
more upmerge and copyrights
tgravescs Jan 10, 2022
aed2d13
upmerge and diffed overrides
tgravescs Jan 10, 2022
352c090
shim Sequence not in 2.3:
tgravescs Jan 10, 2022
6e5ea9a
update copyrights and update RapidsConf
tgravescs Jan 10, 2022
bc4eede
finish copyright updates
tgravescs Jan 10, 2022
eda55f2
building 248 and 232
tgravescs Jan 10, 2022
ddddf35
add pom file
tgravescs Jan 10, 2022
c0346e8
fix style issues
tgravescs Jan 10, 2022
36396f9
fix style
tgravescs Jan 10, 2022
7282c0c
remove shim layers
tgravescs Jan 11, 2022
03610be
update pom description
tgravescs Jan 11, 2022
84fd72f
Add CheckOverflow
tgravescs Jan 11, 2022
70ff0bf
fix line length
tgravescs Jan 11, 2022
233b259
fix copyright
tgravescs Jan 11, 2022
6402631
Merge remote-tracking branch 'origin/branch-22.02' into explainonlymode
tgravescs Jan 12, 2022
13d7ee1
cleanup
tgravescs Jan 12, 2022
ab49521
Update to use the spark.rapids.sql.mode config
tgravescs Jan 12, 2022
ce38223
update config doc and formatting
tgravescs Jan 12, 2022
b1dd70e
update docs
tgravescs Jan 12, 2022
c94d732
fix typo
tgravescs Jan 12, 2022
0003e4e
update configs.md
tgravescs Jan 12, 2022
241f62c
change pom to 24x
tgravescs Jan 12, 2022
6ee3c0c
Merge remote-tracking branch 'origin/branch-22.02' into spark2shim-se…
tgravescs Jan 13, 2022
bb87652
Upmerge some files
tgravescs Jan 13, 2022
1340d9a
change name of jar
tgravescs Jan 13, 2022
77c82a2
Change to check for isSqlEnabled and the mode separately because we may
tgravescs Jan 13, 2022
c02dae5
fix spacing
tgravescs Jan 13, 2022
2afeb2b
update auto generated configs doc
tgravescs Jan 13, 2022
4c98c6e
Add ExplainPlan
tgravescs Jan 13, 2022
f5dc619
Merge remote-tracking branch 'tgravescs/explainonlymode' into spark2s…
tgravescs Jan 13, 2022
1eca823
Update docs for 2.x explain api
tgravescs Jan 13, 2022
ac9049a
cleanup
tgravescs Jan 13, 2022
6c06b31
update build scripts to build 24x
tgravescs Jan 13, 2022
1dc83df
remove unneeded tag
tgravescs Jan 13, 2022
48ae9fd
start diff scripts
tgravescs Jan 13, 2022
85088fa
checkpoint
tgravescs Jan 13, 2022
935c25e
checkpoint
tgravescs Jan 13, 2022
e7c7a0b
checkpoint
tgravescs Jan 13, 2022
cf55dc6
checkpoint
tgravescs Jan 14, 2022
359c306
checkpoint
tgravescs Jan 14, 2022
5b1d40a
checkpoint
tgravescs Jan 14, 2022
cbf62a5
add more
tgravescs Jan 14, 2022
126919a
checkpoint
tgravescs Jan 14, 2022
75edb2d
more diffs done
tgravescs Jan 14, 2022
0db3a3d
more diffs
tgravescs Jan 14, 2022
8bb8291
diff files
tgravescs Jan 14, 2022
b7aa8c7
split apart some functions for easier diff
tgravescs Jan 14, 2022
2dc5251
Overrides diffs and split apart for easier diffing
tgravescs Jan 14, 2022
8b5d720
more diffs and move
tgravescs Jan 14, 2022
8907fac
update script to tmp dir
tgravescs Jan 14, 2022
abd12b0
builds
tgravescs Jan 14, 2022
7437f30
Merge remote-tracking branch 'origin/branch-22.02' into spark2shim-se…
tgravescs Jan 14, 2022
8b40694
Upmerge pull changes
tgravescs Jan 14, 2022
de98f8b
update diff file
tgravescs Jan 14, 2022
9933cdd
Finish diffing the rest of execs, split one out to another file for
tgravescs Jan 14, 2022
82d5818
Merge remote-tracking branch 'origin/branch-22.02' into spark2shim-se…
tgravescs Jan 14, 2022
b912fc6
Update spark2 RapidsConf with changes in sql-plugin
tgravescs Jan 14, 2022
9116768
Update RapidsConf to deal with toBytes different 2.x
tgravescs Jan 14, 2022
52447c8
cleanup documentation
tgravescs Jan 14, 2022
f3c2636
Update docs/get-started/getting-started-workload-qualification.md
tgravescs Jan 18, 2022
f1c55d2
copyright, remove 2.3 from DateUtils, fix up some comments and remove
tgravescs Jan 18, 2022
3bfde90
Merge branch 'spark2shim-sep-module-upmerge' of github.com:tgravescs/…
tgravescs Jan 18, 2022
6f190a6
Update RapidsMeta comment and diff
tgravescs Jan 18, 2022
08b2490
Update 2.x to sql-plugin diffs
tgravescs Jan 18, 2022
b724245
add deploy of new jar for nightly
tgravescs Jan 19, 2022
33da75b
Remove unneeded dependencies from pom file, update NOTICE pulled, remove
tgravescs Jan 19, 2022
0bad0cc
remove python files
tgravescs Jan 19, 2022
f532d81
Merge remote-tracking branch 'origin/branch-22.02' into spark2shim-se…
tgravescs Jan 19, 2022
008d88b
Upmerged to latest sql-plugin code
tgravescs Jan 19, 2022
cc97619
update NOTICE copy and remove transient
tgravescs Jan 19, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix spacing
tgravescs committed Jan 13, 2022

Verified

This commit was signed with the committer’s verified signature.
radoering Randy Döring
commit c02dae5eecf64d686a6aedd0807449c48950102e
Original file line number Diff line number Diff line change
@@ -506,7 +506,7 @@ object RapidsConf {
.createWithDefault(true)

val SQL_MODE = conf("spark.rapids.sql.mode")
.doc("Set the mode for the plugin. The supported modes are explainOnly and executeOnGPU." +
.doc("Set the mode for the plugin. The supported modes are explainOnly and executeOnGPU. " +
"This config can not be changed at runtime, you must restart the application for it " +
"to take affect. The default mode is executeOnGPU, which means the sql plugin will " +
"convert the Spark operations and execute them on the GPU when possible. The " +