[BUG] fastparquet tests fail on Databricks 13.3 due to NaNs becoming nulls when converting from pandas

[jlowe]

On Databricks 13.3, nulls in the Pandas DataFrame (represented as NaNs) are being honored as nulls in the resulting Spark DataFrame when converting a Pandas DataFrame to a Spark DataFrame. Pandas thinks there are nulls in the data, and those nulls are propagating to the Spark DataFrame.

fastparquet loads the NaNs properly, but then when converting the data to pandas, pandas thinks the NaN values are null. This, in turn, causes spark.createDataFrame to produce corresponding nulls. When comparing this to the GPU direct load of the data that contains NaNs (not nulls), the test fails. The problem is not in the way the GPU loads the data, it's the way the NaNs get converted into nulls due to sending the data through pandas before converting to a Spark DataFrame.

[sameerz]

Duplicate of #9776

[mythrocks]

Duplicate of #9776

Sorry, no, it's not. #9776 is for timestamps. This failure is for floating point types.

This is a dupe of #9767, though. And it's been xfailed as part of #9677.