[DOC] behavior of spark.rapids.sql.enabled

[revans2]

Report needed documentation

Report needed documentation
If I created a dataframe (without executing) I can't change that the plugin is enabled or disabled after the fact (with the same df reference). I can only get the change to take effect after recreating df.

It would be good to document why this is happening, so the user can plan for it.

With the plugin enabled:

scala> df.explain
19/09/24 16:18:37 WARN GpuOverrides: *Exec <HashAggregateExec> could run on GPU
*Expression <AggregateExpression> sum(sum_quantity#32) could run on GPU *Expression <Sum> sum(sum_quantity#32) could run on GPU
....
== Physical Plan ==
*(1) GpuColumnarToRow
...

With the plugin disabled, df still shows a GPU plan:

scala> spark.conf.set("spark.rapids.sql.enabled", false)
scala> df.explain
== Physical Plan ==
*(1) GpuColumnarToRow
...

If I recreate it, we are back to the CPU:

scala> val df = spark.sql(q)
df: org.apache.spark.sql.DataFrame = [l_linenumber: bigint, sum(sum_quantity): double ... 2 more fields]
scala> df.explain
== Physical Plan ==
*(3) HashAggregate(keys=[l_linenumber#3L], functions=[sum(sum_quantity#76), min(min_quantity#77), max(max_quantity#78)])
...

[revans2]

The reason for this is that executedPlan in DataSet is a lazy val. The explain is the only thing that is impacted by this. Actual execution happens as expected.