Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] SPARK UT Framework showing tests passed, but Exception printed without failing UT. #11432

Open
Feng-Jiang28 opened this issue Sep 6, 2024 · 0 comments
Labels
bug Something isn't working test Only impacts tests

Comments

@Feng-Jiang28
Copy link
Contributor

Feng-Jiang28 commented Sep 6, 2024

Describe the bug
When working on enbling Parquet Unit tests on ParquetFieldIdIOSuite(#11366), it shows that all 5 tests have passed, but in the ternimal you can see that, the tests of "read parquet file ids" and "global read/write flag should work correctly" throws java.lang.RuntimeExceptions, without stopping the UT.

24/09/06 10:39:29.802 Executor task launch worker for task 1.0 in stage 42.0 (TID 79) ERROR Executor: Exception in task 1.0 in stage 42.0 (TID 79)
java.lang.RuntimeException:  Found duplicate field(s) "1": [a, rand2] in case-insensitive mode                  
        at com.nvidia.spark.rapids.ParquetSchemaUtils$.$anonfun$clipParquetGroupFields$8(ParquetSchemaUtils.scala:310) ~[rapids-4-spark-aggregator_2.12-24.10.0-SNAPSHOT-spark330.jar:?]
        at scala.Option.map(Option.scala:230) ~[scala-library-2.12.15.jar:?]
        at com.nvidia.spark.rapids.ParquetSchemaUtils$.matchIdField$1(ParquetSchemaUtils.scala:302) ~[rapids-4-spark-aggregator_2.12-24.10.0-SNAPSHOT-spark330.jar:?]
        at com.nvidia.spark.rapids.ParquetSchemaUtils$.$anonfun$clipParquetGroupFields$10(ParquetSchemaUtils.scala:320) ~[rapids-4-spark-aggregator_2.12-24.10.0-SNAPSHOT-spark330.jar:?]
        at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293) ~[scala-library-2.12.15.jar:?]
        at scala.collection.Iterator.foreach(Iterator.scala:943) ~[scala-library-2.12.15.jar:?]
        at scala.collection.Iterator.foreach$(Iterator.scala:943) ~[scala-library-2.12.15.jar:?]

Steps/Code to reproduce bug
Checkout PR (#11366),
comment RapidsTestSettings.scala in tests/src/test/spark330

.exclude("read parquet file without ids", ADJUST_UT("Showing that the test has passed, but there is RuntimeException thrown"))
.exclude("global read/write flag should work correctly", ADJUST_UT("Showing that the test has passed, but there is RuntimeException thrown"))

compile and run the test with
mvn -nsu -Dbuildver=330 -pl tests -Dsuites="org.apache.spark.sql.rapids.suites.RapidsParquetFieldIdSchemaSuite" test

Expected behavior
If Exception thrown, UT should fail.

@Feng-Jiang28 Feng-Jiang28 added ? - Needs Triage Need team to review and classify bug Something isn't working labels Sep 6, 2024
@Feng-Jiang28 Feng-Jiang28 changed the title [BUG] SPARK UT Framework showing test passed, but Exception printed without failing UT. [BUG] SPARK UT Framework showing tests passed, but Exception printed without failing UT. Sep 6, 2024
@GaryShen2008 GaryShen2008 added the test Only impacts tests label Sep 6, 2024
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Oct 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working test Only impacts tests
Projects
None yet
Development

No branches or pull requests

3 participants