Implement optimized AQE support so that exchanges run on GPU where possible #462

andygrove · 2020-07-29T15:14:12Z

This PR implements optimized AQE support for Spark 3.0.1 and 3.1.0, where shuffle and broadcast exchanges stay on the GPU where supported.

andygrove · 2020-07-29T19:47:54Z

build

Signed-off-by: Andy Grove <[email protected]>

andygrove · 2020-07-31T22:56:35Z

build

tests/src/test/scala/com/nvidia/spark/rapids/StringFallbackSuite.scala

Signed-off-by: Andy Grove <[email protected]>

andygrove · 2020-08-03T15:14:41Z

build

shims/spark310/src/main/scala/com/nvidia/spark/rapids/shims/spark310/GpuHashJoin.scala

.../spark300/src/main/scala/com/nvidia/spark/rapids/shims/spark300/GpuShuffleExchangeExec.scala

...spark300/src/main/scala/org/apache/spark/sql/rapids/shims/spark300/ShuffleManagerShims.scala

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/ShuffleManagerShims.scala

...park301/src/main/scala/com/nvidia/spark/rapids/shims/spark301/GpuBroadcastExchangeExec.scala

sql-plugin/src/main/scala/com/nvidia/spark/rapids/SparkShims.scala

sql-plugin/src/main/scala/com/nvidia/spark/rapids/limit.scala

...plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuCustomShuffleReaderExec.scala

tests/src/test/scala/com/nvidia/spark/rapids/StringFallbackSuite.scala

Signed-off-by: Andy Grove <[email protected]>

andygrove · 2020-08-03T16:46:33Z

This PR also closes #492

Signed-off-by: Andy Grove <[email protected]>

abellina · 2020-08-07T21:51:16Z

...park301/src/main/scala/com/nvidia/spark/rapids/shims/spark301/GpuBroadcastHashJoinExec.scala

+      }
+
+      if (!canThisBeReplaced) {
+       buildSide.willNotWorkOnGpu("the BroadcastHashJoin this feeds is not on the GPU")


abellina · 2020-08-08T12:42:37Z

I’m not seeing anything big stand out, just a couple of nits above. Thanks @andygrove

Signed-off-by: Andy Grove <[email protected]>

andygrove · 2020-08-10T15:08:00Z

build

Signed-off-by: Andy Grove <[email protected]>

andygrove · 2020-08-10T15:30:40Z

build

Signed-off-by: Andy Grove <[email protected]>

andygrove · 2020-08-10T15:59:06Z

build

andygrove · 2020-08-10T16:54:23Z

@tgravescs @abellina I believe all issues are addressed. I have also enabled integration tests for TPC-H Q2 with AQE now that the GpuFilter fix has been merged.

Signed-off-by: Andy Grove <[email protected]>

andygrove · 2020-08-12T17:40:51Z

build

tgravescs

I don't think we actually need all the classes with ShuffleManagerShimBase as we could just have a function in Spark300Shims, but I think its fine as they are actively messing with Shuffle, so something else is bound to change there

...spark300/src/main/scala/org/apache/spark/sql/rapids/shims/spark300/ShuffleManagerShims.scala

tests/src/test/scala/com/nvidia/spark/rapids/SparkQueryCompareTestSuite.scala

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuBroadcastExchangeExec.scala

andygrove · 2020-08-13T20:50:42Z

build

tgravescs

looks good pending Jenkins

Changes were addressed.

…ssible (NVIDIA#462) * Implement optimized support for AQE Signed-off-by: Andy Grove <[email protected]> * revert scalastyle config change Signed-off-by: Andy Grove <[email protected]> * prep for review Signed-off-by: Andy Grove <[email protected]> * remove temp debug println Signed-off-by: Andy Grove <[email protected]> * enable AQE tests Signed-off-by: Andy Grove <[email protected]> * fix regression with 3.1.0 Signed-off-by: Andy Grove <[email protected]> * address some formatting issues from PR review Signed-off-by: Andy Grove <[email protected]> * address some formatting issues from PR review Signed-off-by: Andy Grove <[email protected]> * resolve TODO comment and more formatting fixes Signed-off-by: Andy Grove <[email protected]> * remove code duplication Signed-off-by: Andy Grove <[email protected]> * remove blank line Signed-off-by: Andy Grove <[email protected]> * address some style feedback Signed-off-by: Andy Grove <[email protected]> * fix odd indenting in one file Signed-off-by: Andy Grove <[email protected]> * updated configs doc Signed-off-by: Andy Grove <[email protected]> * revert format changes to optimizeGpuPlanTransitions Signed-off-by: Andy Grove <[email protected]> * run python tpch tests with aqe on and off Signed-off-by: Andy Grove <[email protected]> * enable tpch query 2 test in scala and python Signed-off-by: Andy Grove <[email protected]> * Revert "enable tpch query 2 test in scala and python" This reverts commit bcd9783. Signed-off-by: Andy Grove <[email protected]> * fix indent Signed-off-by: Andy Grove <[email protected]> * enable AQE testing for TPC-H query 2 Signed-off-by: Andy Grove <[email protected]> * fix error in python test Signed-off-by: Andy Grove <[email protected]> * rename two source files and remove a blank linke

Signed-off-by: spark-rapids automation <[email protected]>

andygrove added the performance A performance related task/issue label Jul 29, 2020

andygrove added this to the Jul 20 - Jul 31 milestone Jul 29, 2020

andygrove mentioned this pull request Jul 30, 2020

[WIP] [POC] Enable broadcast and shuffle exchanges to stay on GPU when AQE is enabled #361

Closed

andygrove changed the title ~~[WIP] Rebasing AQE work on SPARK-32332 and shim layer~~ [WIP] Implement optimized AQE support so that exchanges run on GPU where possible Jul 31, 2020

Implement optimized support for AQE

85a709b

Signed-off-by: Andy Grove <[email protected]>

andygrove force-pushed the adaptive-query-SPARK-32332 branch from 9a3b53f to 85a709b Compare July 31, 2020 16:41

revert scalastyle config change

b25723d

Signed-off-by: Andy Grove <[email protected]>

andygrove mentioned this pull request Jul 31, 2020

[BUG] Re-enable AQE integration tests #492

Closed

Merge branch 'branch-0.2' into adaptive-query-SPARK-32332

b6ade68

Signed-off-by: Andy Grove <[email protected]>

andygrove mentioned this pull request Jul 31, 2020

[FEA] Implement shuffle optimization when AQE is enabled #493

Closed

andygrove changed the title ~~[WIP] Implement optimized AQE support so that exchanges run on GPU where possible~~ Implement optimized AQE support so that exchanges run on GPU where possible Jul 31, 2020

andygrove commented Jul 31, 2020

View reviewed changes

tests/src/test/scala/com/nvidia/spark/rapids/StringFallbackSuite.scala Outdated Show resolved Hide resolved

andygrove added 5 commits August 3, 2020 07:32

merge from branch-0.2

14f5593

Signed-off-by: Andy Grove <[email protected]>

prep for review

40a082e

Signed-off-by: Andy Grove <[email protected]>

remove temp debug println

9c072a1

Signed-off-by: Andy Grove <[email protected]>

enable AQE tests

e6fbaef

Signed-off-by: Andy Grove <[email protected]>

fix regression with 3.1.0

1dc94f0

Signed-off-by: Andy Grove <[email protected]>

tgravescs reviewed Aug 3, 2020

View reviewed changes

address some formatting issues from PR review

34eba7b

Signed-off-by: Andy Grove <[email protected]>

andygrove modified the milestones: Jul 20 - Jul 31, Aug 3 - Aug 14 Aug 3, 2020

address some formatting issues from PR review

0ab4f98

Signed-off-by: Andy Grove <[email protected]>

andygrove added 3 commits August 3, 2020 10:57

resolve TODO comment and more formatting fixes

ce43515

Signed-off-by: Andy Grove <[email protected]>

remove code duplication

9883221

Signed-off-by: Andy Grove <[email protected]>

Merge branch 'branch-0.2' into adaptive-query-SPARK-32332

7556d19

Signed-off-by: Andy Grove <[email protected]>

abellina reviewed Aug 7, 2020

View reviewed changes

fix indent

d10d60b

Signed-off-by: Andy Grove <[email protected]>

andygrove added 2 commits August 10, 2020 09:14

enable AQE testing for TPC-H query 2

e29db29

Signed-off-by: Andy Grove <[email protected]>

Merge branch 'branch-0.2' into adaptive-query-SPARK-32332

9c68e76

Signed-off-by: Andy Grove <[email protected]>

fix error in python test

4d6f823

Signed-off-by: Andy Grove <[email protected]>

andygrove mentioned this pull request Aug 10, 2020

Add test for skewed join optimization when AQE is enabled #536

Merged

sameerz mentioned this pull request Aug 11, 2020

[FEA] Update HashSortOptimizerSuite to work with AQE #452

Closed

Merge branch 'branch-0.2' into adaptive-query-SPARK-32332

9dbd02a

Signed-off-by: Andy Grove <[email protected]>

tgravescs reviewed Aug 13, 2020

View reviewed changes

rename two source files and remove a blank linke

7f2c76a

tgravescs approved these changes Aug 13, 2020

View reviewed changes

andygrove merged commit b86fd32 into NVIDIA:branch-0.2 Aug 13, 2020

andygrove deleted the adaptive-query-SPARK-32332 branch August 13, 2020 21:44

tgravescs pushed a commit to tgravescs/spark-rapids that referenced this pull request Nov 30, 2023

Update submodule cudf to acadcf2 (NVIDIA#462)

622790c

Signed-off-by: spark-rapids automation <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement optimized AQE support so that exchanges run on GPU where possible #462

Implement optimized AQE support so that exchanges run on GPU where possible #462

andygrove commented Jul 29, 2020 •

edited

Loading

andygrove commented Jul 29, 2020

andygrove commented Jul 31, 2020

andygrove commented Aug 3, 2020

andygrove commented Aug 3, 2020

abellina Aug 7, 2020

abellina commented Aug 8, 2020

andygrove commented Aug 10, 2020

andygrove commented Aug 10, 2020

andygrove commented Aug 10, 2020

andygrove commented Aug 10, 2020

andygrove commented Aug 12, 2020

tgravescs left a comment

andygrove commented Aug 13, 2020

tgravescs left a comment

Implement optimized AQE support so that exchanges run on GPU where possible #462

Implement optimized AQE support so that exchanges run on GPU where possible #462

Conversation

andygrove commented Jul 29, 2020 • edited Loading

andygrove commented Jul 29, 2020

andygrove commented Jul 31, 2020

andygrove commented Aug 3, 2020

andygrove commented Aug 3, 2020

abellina Aug 7, 2020

Choose a reason for hiding this comment

abellina commented Aug 8, 2020

andygrove commented Aug 10, 2020

andygrove commented Aug 10, 2020

andygrove commented Aug 10, 2020

andygrove commented Aug 10, 2020

andygrove commented Aug 12, 2020

tgravescs left a comment

Choose a reason for hiding this comment

andygrove commented Aug 13, 2020

tgravescs left a comment

Choose a reason for hiding this comment

andygrove commented Jul 29, 2020 •

edited

Loading