[BUG] Unit test failures in Spark 3.2 shim build

[gerashegalov]

Describe the bug
Umbrella issue to fix unit tests in the 3.2 shim

Steps/Code to reproduce bug

$ mvn clean package -Dbuildver=320 -Dskip  

Expected behavior
no unit test failures

Environment details (please complete the following information)
local

Additional context
#3323

Tasks:

HashAggregatesSuite

Fixed by #3403

• WITH DECIMALS: Avg Distinct with filter *** FAILED ***

CastOpSuite

See #3382 for more detail.

Fixed by #3439 and #3531

• Cast from string to date using random inputs *** FAILED ***
• Cast from string to date using random inputs with valid year prefix *** FAILED ***
• Cast from string to timestamp *** FAILED ***
• Test all supported casts with in- [ ] range values *** FAILED ***

ExpandExecSuite

Fixed by #3403

• IGNORE ORDER, WITH DECIMALS: group with aggregates *** FAILED ***

WindowFunctionSuite

Filed as #3415

Fixed by #3547

• WITH DECIMALS: [Window] [RANGE] [ ASC] [- [ ] 2 DAYS, 3 DAYS] *** FAILED ***
• WITH DECIMALS: [Window] [RANGE] [DESC] [- [ ] 2 DAYS, 3 DAYS] *** FAILED ***
• WITH DECIMALS: [Window] [RANGE] [ ASC] [- [ ] 2 DAYS, CURRENT ROW] *** FAILED ***
• WITH DECIMALS: [Window] [RANGE] [DESC] [- [ ] 2 DAYS, CURRENT ROW] *** FAILED ***
• WITH DECIMALS: [Window] [RANGE] [ ASC] [CURRENT ROW, 3 DAYS] *** FAILED ***
• WITH DECIMALS: [Window] [RANGE] [DESC] [CURRENT ROW, 3 DAYS] *** FAILED ***
• IGNORE ORDER, WITH DECIMALS: [Window] [MIXED WINDOW SPECS] *** FAILED ***

ParquetWriterSuite

These tests were fixed by #3377

• Job commit time metrics *** FAILED ***

CostBasedOptimizerSuite

These tests were fixed by #3377 and SPARK-36666

• keep CustomShuffleReaderExec on GPU *** FAILED ***
• Compute estimated row count nested joins no broadcast *** FAILED ***

These tests need updating and have been ignored for now. There is a follow-on issue #3526 to update them when we resume work on CBO.

• Avoid transition to GPU for trivial projection after CPU SMJ *** FAILED ***
• Force section of plan back onto CPU, AQE on *** FAILED ***
• Force last section of plan back onto CPU, AQE on *** FAILED ***

AdaptiveQueryExecSuite

These tests were fixed by #3377 and SPARK-36666

• Plugin should translate child plan of GPU DataWritingCommandExec to GPU *** FAILED ***
• Plugin should translate child plan of CPU DataWritingCommandExec to GPU *** FAILED ***
• Exchange reuse *** FAILED ***
• Change merge join to broadcast join without local shuffle reader *** FAILED ***
• Verify the reader is LocalShuffleReaderExec *** FAILED ***
• Avoid splitting CustomShuffleReader and ShuffleQueryStageExec pairs *** FAILED ***

KnownNotNullSuite

Fixed by #3457

• KnownNotNull in Scala UDF- [ ] Int *** FAILED ***
• KnownNotNull in Scala UDF- [ ] Long *** FAILED ***
• KnownNotNull in Scala UDF- [ ] Short *** FAILED ***

AnsiCastOpSuite

These tests were fixed by #3377

• WITH DECIMALS: Write bytes to string *** FAILED ***
• WITH DECIMALS: Write shorts to string *** FAILED ***
• WITH DECIMALS: Write ints to string *** FAILED ***
• WITH DECIMALS: Write longs to string *** FAILED ***
• WITH DECIMALS: Write ints to long *** FAILED ***
• WITH DECIMALS: Write longs to int (values within range) *** FAILED ***
• WITH DECIMALS: Write longs to short (values within range) *** FAILED ***
• WITH DECIMALS: Write longs to byte (values within range) *** FAILED ***
• WITH DECIMALS: Write ints to short (values within range) *** FAILED ***
• WITH DECIMALS: Write ints to byte (values within range) *** FAILED ***
• WITH DECIMALS: Write shorts to byte (values within range) *** FAILED ***
• WITH DECIMALS: Write floats to long (values within range) *** FAILED ***
• WITH DECIMALS: Write floats to int (values within range) *** FAILED ***
• WITH DECIMALS: Write floats to short (values within range) *** FAILED ***
• WITH DECIMALS: Write floats to byte (values within range) *** FAILED ***
• WITH DECIMALS: Write doubles to long (values within range) *** FAILED ***
• WITH DECIMALS: Write doubles to int (values within range) *** FAILED ***
• WITH DECIMALS: Write doubles to short (values within range) *** FAILED ***
• WITH DECIMALS: Write doubles to byte (values within range) *** FAILED ***
• WITH DECIMALS: Copy ints to long *** FAILED ***
• WITH DECIMALS: Copy long to float *** FAILED ***

ParseDateTimeSuite

See #3383 for more detail. fixed by #3439

• WITH DECIMALS: to_date yyyy- [ ] MM- [ ] dd *** FAILED ***
• WITH DECIMALS: to_date dd/MM/yyyy *** FAILED ***
• WITH DECIMALS: to_date default pattern *** FAILED ***
• parse now *** FAILED ***

ScalarSubquerySuite

Failures due to #3400 (canonicalizationMatchesCpu=true != canonicalizationMatchesGpu=false)

• WITH DECIMALS: Uncorrelated Scalar Subquery *** FAILED ***

[andygrove]

I am looking at the AnsiCastOpSuite failures with the pattern WITH DECIMALS: Write _ to _.

[andygrove]

I also started looking at CastOpSuite and filed a separate issue since this looks like it could be quite involved. #3382

[andygrove]

I filed #3383 for ParseDateTimeSuite failures

[tgravescs]

cast ops is fixed by 3531 and no other unit tests failures currently

[tgravescs]

3.2 unit tests all pass now