[BUG] test_regexp_replace[DATAGEN_SEED=1722297411, TZ=UTC] hanging there forever in pre-merge CI intermittently

[pxLi]

Describe the bug
first seen in CI of a non-related change
#11268 (comment)

test_regexp_replace[DATAGEN_SEED=1722297411, TZ=UTC]

rapids_premerge-github run:9834
https://github.com/NVIDIA/spark-rapids/blob/branch-24.08/jenkins/spark-premerge-build.sh#L223-L224

executor was hanging there making no progress,

[INFO] 2024-07-29 23:57:06,351 org.sparkproject.jetty.util.log initialized - Logging initialized @13791ms to org.sparkproject.jetty.util.log.Slf4jLog
2024-07-29 23:57:11 INFO     Running test 'src/main/python/regexp_test.py::test_split_regexp_disabled_fallback[DATAGEN_SEED=1722297411, TZ=UTC, INJECT_OOM, ALLOW_NON_GPU(ProjectExec,StringSplit)]'
[WARN] 2024-07-29 23:57:17,340 com.nvidia.spark.rapids.GpuOverrides logWarning - 
!Exec <ProjectExec> cannot run on GPU because not all expressions can be replaced
  @Expression <Alias> split(a#16, [:], 2) AS split(a, [:], 2)#18 could run on GPU
    !Expression <StringSplit> split(a#16, [:], 2) cannot run on GPU because regular expression support is disabled. Set spark.rapids.sql.regexp.enabled=true to enable it
      @Expression <AttributeReference> a#16 could run on GPU
      @Expression <Literal> [:] could run on GPU
      @Expression <Literal> 2 could run on GPU
  @Expression <Alias> split(a#16, [o:], 5) AS split(a, [o:], 5)#19 could run on GPU
    !Expression <StringSplit> split(a#16, [o:], 5) cannot run on GPU because regular expression support is disabled. Set spark.rapids.sql.regexp.enabled=true to enable it
      @Expression <AttributeReference> a#16 could run on GPU
      @Expression <Literal> [o:] could run on GPU
      @Expression <Literal> 5 could run on GPU
  @Expression <Alias> split(a#16, [^:], 2) AS split(a, [^:], 2)#20 could run on GPU
    !Expression <StringSplit> split(a#16, [^:], 2) cannot run on GPU because regular expression support is disabled. Set spark.rapids.sql.regexp.enabled=true to enable it
      @Expression <AttributeReference> a#16 could run on GPU
      @Expression <Literal> [^:] could run on GPU
      @Expression <Literal> 2 could run on GPU
  @Expression <Alias> split(a#16, [^o], 55) AS split(a, [^o], 55)#21 could run on GPU
    !Expression <StringSplit> split(a#16, [^o], 55) cannot run on GPU because regular expression support is disabled. Set spark.rapids.sql.regexp.enabled=true to enable it
      @Expression <AttributeReference> a#16 could run on GPU
      @Expression <Literal> [^o] could run on GPU
      @Expression <Literal> 55 could run on GPU
  @Expression <Alias> split(a#16, [o]{1,2}, 999) AS split(a, [o]{1,2}, 999)#22 could run on GPU
    !Expression <StringSplit> split(a#16, [o]{1,2}, 999) cannot run on GPU because regular expression support is disabled. Set spark.rapids.sql.regexp.enabled=true to enable it
      @Expression <AttributeReference> a#16 could run on GPU
      @Expression <Literal> [o]{1,2} could run on GPU
      @Expression <Literal> 999 could run on GPU
  @Expression <Alias> split(a#16, [bf], 2) AS split(a, [bf], 2)#23 could run on GPU
    !Expression <StringSplit> split(a#16, [bf], 2) cannot run on GPU because regular expression support is disabled. Set spark.rapids.sql.regexp.enabled=true to enable it
      @Expression <AttributeReference> a#16 could run on GPU
      @Expression <Literal> [bf] could run on GPU
      @Expression <Literal> 2 could run on GPU
  @Expression <Alias> split(a#16, [o], 5) AS split(a, [o], 5)#24 could run on GPU
    !Expression <StringSplit> split(a#16, [o], 5) cannot run on GPU because regular expression support is disabled. Set spark.rapids.sql.regexp.enabled=true to enable it
      @Expression <AttributeReference> a#16 could run on GPU
      @Expression <Literal> [o] could run on GPU
      @Expression <Literal> 5 could run on GPU
  ! <RDDScanExec> cannot run on GPU because GPU does not currently support the operator class org.apache.spark.sql.execution.RDDScanExec
    @Expression <AttributeReference> a#16 could run on GPU

2024-07-29 23:57:17 INFO     Running test 'src/main/python/regexp_test.py::test_split_escaped_chars_in_character_class[DATAGEN_SEED=1722297411, TZ=UTC]'
[WARN] 2024-07-29 23:57:18,967 com.nvidia.spark.rapids.GpuOverrides logWarning - 
  ! <RDDScanExec> cannot run on GPU because GPU does not currently support the operator class org.apache.spark.sql.execution.RDDScanExec
    @Expression <AttributeReference> a#48 could run on GPU

2024-07-29 23:57:19 INFO     Running test 'src/main/python/regexp_test.py::test_regexp_replace[DATAGEN_SEED=1722297411, TZ=UTC]'
[WARN] 2024-07-29 23:57:20,955 com.nvidia.spark.rapids.GpuOverrides logWarning - 
  ! <RDDScanExec> cannot run on GPU because GPU does not currently support the operator class org.apache.spark.sql.execution.RDDScanExec
    @Expression <AttributeReference> a#76 could run on GPU

Steps/Code to reproduce bug
not always reproducible.

Expected behavior
A clear and concise description of what you expected to happen.

Environment details (please complete the following information)

• Environment location: [Standalone, YARN, Kubernetes, Cloud(specify cloud provider)]
• Spark configuration settings related to the issue

Additional context
Add any other context about the problem here.

[pxLi]

could also be another instance of #11265

will keep monitoring to see if any new report

[pxLi]

no report after #11265 fix. closing this