Skip to content

Commit

Permalink
Add comments for lazy binding in WindowInPandas (NVIDIA#2496)
Browse files Browse the repository at this point in the history
Signed-off-by: Firestarman <[email protected]>
  • Loading branch information
firestarman authored May 25, 2021
1 parent 558d8d8 commit 5a2eb56
Show file tree
Hide file tree
Showing 2 changed files with 9 additions and 1 deletion.
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2020, NVIDIA CORPORATION.
* Copyright (c) 2020-2021, NVIDIA CORPORATION.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand Down Expand Up @@ -54,6 +54,10 @@ case class GpuWindowInPandasExec(
}
}

// On Databricks, binding the references on driver side will get some invalid expressions
// (e.g. none#0L, none@1L) in the `projectList`, causing failures in `test_window` test.
// So need to do the binding for `projectList` lazily, and the binding will actually run
// on executors now.
private lazy val outReferences = {
val allExpressions = windowFramesWithExpressions.map(_._2).flatten
val references = allExpressions.zipWithIndex.map { case (e, i) =>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,10 @@ case class GpuWindowInPandasExec(
}
}

// On Databricks, binding the references on driver side will get some invalid expressions
// (e.g. none#0L, none@1L) in the `projectList`, causing failures in `test_window` test.
// So need to do the binding for `projectList` lazily, and the binding will actually run
// on executors now.
private lazy val outReferences = {
val allExpressions = windowFramesWithExpressions.map(_._2).flatten
val references = allExpressions.zipWithIndex.map { case (e, i) =>
Expand Down

0 comments on commit 5a2eb56

Please sign in to comment.