Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Compile failure on Databricks 11.3 with GpuHiveTableScanExec.scala #8460

Closed
NVnavkumar opened this issue May 31, 2023 · 0 comments · Fixed by #8462
Closed

[BUG] Compile failure on Databricks 11.3 with GpuHiveTableScanExec.scala #8460

NVnavkumar opened this issue May 31, 2023 · 0 comments · Fixed by #8462
Assignees
Labels
bug Something isn't working build Related to CI / CD or cleanly building

Comments

@NVnavkumar
Copy link
Collaborator

The plugin fails to compile on branch-23.06 fails to compile on Databricks 11.3 with the following:

[WARNING] Pruning sources from previous analysis, due to incompatible CompileSetup.
[INFO] Compiling 386 Scala sources and 59 Java sources to /home/ubuntu/spark-rapids/sql-plugin/target/spark330db/classes...
[ERROR] /home/ubuntu/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/hive/rapids/GpuHiveTableScanExec.scala:274: type mismatch;
 found   : org.apache.spark.sql.execution.datasources.SerializableFileStatus
 required: org.apache.hadoop.fs.FileStatus
[ERROR]           isSplitable = canBeSplit(f, hadoopConf),
[ERROR]                                    ^
[ERROR] /home/ubuntu/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/hive/rapids/GpuHiveTableScanExec.scala:281: type mismatch;
 found   : Array[Nothing]
 required: Seq[org.apache.spark.sql.execution.datasources.PartitionedFile]
[ERROR]     val filePartitions = FilePartition.getFilePartitions(sparkSession, splitFiles, maxSplitBytes)
[ERROR]                                                                        ^
[ERROR] two errors found

Might be related to fixes from #8420

@NVnavkumar NVnavkumar added bug Something isn't working ? - Needs Triage Need team to review and classify build Related to CI / CD or cleanly building labels May 31, 2023
@mythrocks mythrocks self-assigned this May 31, 2023
@mythrocks mythrocks removed the ? - Needs Triage Need team to review and classify label May 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working build Related to CI / CD or cleanly building
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants