Fix degenerate conditional nested loop join detection [databricks]

[jlowe]

Fixes #11266. The failure on Databricks is because the aggregate was pushed through the join, which resulted in a non-empty output from the join. The fix from #11244 was flawed in that it detected unconditional joins if the condition was true and the output is empty (i.e.: a row-count-only join), but this last condition isn't necessary. Nested loop joins are unconditional joins if the join condition is always true regardless of the outputs being produced by the join.

[jlowe]

build

[jlowe]

CI failed in test_right_broadcast_nested_loop_join_without_condition_empty, which exposed that we were not properly handling empty build-side batches in unconditional nested outer loop joins. Previously we hacked around it by adding an always-true condition, but this adds support for it and removes the hack.

[jlowe]

build

[revans2]

Do we need to worry about a full outer join?

[jlowe]

In the future yes, but we do not support FullOuter joins for broadcasted nested loop joins. Support for that is tracked by #3269.

[jlowe]

CI failure appears to be related to rapidsai/cudf#16426.

[jlowe]

build

[pxLi]

some regex cases (https://github.com/NVIDIA/spark-rapids/blob/branch-24.08/jenkins/spark-premerge-build.sh#L223-L224) seems to be hanging there forever in scala213 CI, like

test_regexp_replace[DATAGEN_SEED=1722297411, TZ=UTC]

executor stop producing any further logs,

[INFO] 2024-07-29 23:57:06,351 org.sparkproject.jetty.util.log initialized - Logging initialized @13791ms to org.sparkproject.jetty.util.log.Slf4jLog
2024-07-29 23:57:11 INFO     Running test 'src/main/python/regexp_test.py::test_split_regexp_disabled_fallback[DATAGEN_SEED=1722297411, TZ=UTC, INJECT_OOM, ALLOW_NON_GPU(ProjectExec,StringSplit)]'
[WARN] 2024-07-29 23:57:17,340 com.nvidia.spark.rapids.GpuOverrides logWarning - 
!Exec <ProjectExec> cannot run on GPU because not all expressions can be replaced
  @Expression <Alias> split(a#16, [:], 2) AS split(a, [:], 2)#18 could run on GPU
    !Expression <StringSplit> split(a#16, [:], 2) cannot run on GPU because regular expression support is disabled. Set spark.rapids.sql.regexp.enabled=true to enable it
      @Expression <AttributeReference> a#16 could run on GPU
      @Expression <Literal> [:] could run on GPU
      @Expression <Literal> 2 could run on GPU
  @Expression <Alias> split(a#16, [o:], 5) AS split(a, [o:], 5)#19 could run on GPU
    !Expression <StringSplit> split(a#16, [o:], 5) cannot run on GPU because regular expression support is disabled. Set spark.rapids.sql.regexp.enabled=true to enable it
      @Expression <AttributeReference> a#16 could run on GPU
      @Expression <Literal> [o:] could run on GPU
      @Expression <Literal> 5 could run on GPU
  @Expression <Alias> split(a#16, [^:], 2) AS split(a, [^:], 2)#20 could run on GPU
    !Expression <StringSplit> split(a#16, [^:], 2) cannot run on GPU because regular expression support is disabled. Set spark.rapids.sql.regexp.enabled=true to enable it
      @Expression <AttributeReference> a#16 could run on GPU
      @Expression <Literal> [^:] could run on GPU
      @Expression <Literal> 2 could run on GPU
  @Expression <Alias> split(a#16, [^o], 55) AS split(a, [^o], 55)#21 could run on GPU
    !Expression <StringSplit> split(a#16, [^o], 55) cannot run on GPU because regular expression support is disabled. Set spark.rapids.sql.regexp.enabled=true to enable it
      @Expression <AttributeReference> a#16 could run on GPU
      @Expression <Literal> [^o] could run on GPU
      @Expression <Literal> 55 could run on GPU
  @Expression <Alias> split(a#16, [o]{1,2}, 999) AS split(a, [o]{1,2}, 999)#22 could run on GPU
    !Expression <StringSplit> split(a#16, [o]{1,2}, 999) cannot run on GPU because regular expression support is disabled. Set spark.rapids.sql.regexp.enabled=true to enable it
      @Expression <AttributeReference> a#16 could run on GPU
      @Expression <Literal> [o]{1,2} could run on GPU
      @Expression <Literal> 999 could run on GPU
  @Expression <Alias> split(a#16, [bf], 2) AS split(a, [bf], 2)#23 could run on GPU
    !Expression <StringSplit> split(a#16, [bf], 2) cannot run on GPU because regular expression support is disabled. Set spark.rapids.sql.regexp.enabled=true to enable it
      @Expression <AttributeReference> a#16 could run on GPU
      @Expression <Literal> [bf] could run on GPU
      @Expression <Literal> 2 could run on GPU
  @Expression <Alias> split(a#16, [o], 5) AS split(a, [o], 5)#24 could run on GPU
    !Expression <StringSplit> split(a#16, [o], 5) cannot run on GPU because regular expression support is disabled. Set spark.rapids.sql.regexp.enabled=true to enable it
      @Expression <AttributeReference> a#16 could run on GPU
      @Expression <Literal> [o] could run on GPU
      @Expression <Literal> 5 could run on GPU
  ! <RDDScanExec> cannot run on GPU because GPU does not currently support the operator class org.apache.spark.sql.execution.RDDScanExec
    @Expression <AttributeReference> a#16 could run on GPU

2024-07-29 23:57:17 INFO     Running test 'src/main/python/regexp_test.py::test_split_escaped_chars_in_character_class[DATAGEN_SEED=1722297411, TZ=UTC]'
[WARN] 2024-07-29 23:57:18,967 com.nvidia.spark.rapids.GpuOverrides logWarning - 
  ! <RDDScanExec> cannot run on GPU because GPU does not currently support the operator class org.apache.spark.sql.execution.RDDScanExec
    @Expression <AttributeReference> a#48 could run on GPU

2024-07-29 23:57:19 INFO     Running test 'src/main/python/regexp_test.py::test_regexp_replace[DATAGEN_SEED=1722297411, TZ=UTC]'
[WARN] 2024-07-29 23:57:20,955 com.nvidia.spark.rapids.GpuOverrides logWarning - 
  ! <RDDScanExec> cannot run on GPU because GPU does not currently support the operator class org.apache.spark.sql.execution.RDDScanExec
    @Expression <AttributeReference> a#76 could run on GPU

[pxLi]

As the above is the last case of CI, and it cannot be reproduced locally.
filed #11270 to track

Im going to merge this to unblock other changes, thanks