feat: eliminate unnecessary projection. #5366

jackwener · 2023-02-22T07:39:09Z

Which issue does this PR close?

Closes #.

Rationale for this change

What changes are included in this PR?

This PR add a rule to eliminate unnecessary projection.

Are these changes tested?

add unit test.

Are there any user-facing changes?

None

alamb

I found using whitespace blind diff https://github.com/apache/arrow-datafusion/pull/5366/files?w=1 made it easier to see what was changed

I reviewed the code and plan changes and they all look very good to me. Thank you @jackwener

alamb · 2023-02-25T15:21:23Z

datafusion/core/tests/sql/aggregates.rs

-        "| 3              | 3               | 3               | 3             |",
-        "+----------------+-----------------+-----------------+---------------+",
+        "+--------------+---------------+---------------+-------------+",
+        "| COUNT(nanos) | COUNT(micros) | COUNT(millis) | COUNT(secs) |",


this looks like an improvement to me

ursabot · 2023-02-25T15:43:24Z

Benchmark runs are scheduled for baseline = 645428f and contender = 8b92b9b. 8b92b9b is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

avantgardnerio · 2023-02-25T19:48:04Z

Merging this PR seems to have broken two tests in main:




expected:

[
    "Explain [plan_type:Utf8, plan:Utf8]",
    "  Projection: t1.t1_id, t1.t1_name, t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "    Filter: EXISTS (<subquery>) [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "      Subquery: [t1_int:UInt32;N]",
    "        Projection: t1.t1_int [t1_int:UInt32;N]",
    "          Filter: t1.t1_id > t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "            TableScan: t1 [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "      TableScan: t1 projection=[t1_id, t1_name, t1_int] [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
]
actual:

[
    "Explain [plan_type:Utf8, plan:Utf8]",
    "  Filter: EXISTS (<subquery>) [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "    Subquery: [t1_int:UInt32;N]",
    "      Projection: t1.t1_int [t1_int:UInt32;N]",
    "        Filter: t1.t1_id > t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "          TableScan: t1 [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "    TableScan: t1 projection=[t1_id, t1_name, t1_int] [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
]


Left:  ["Explain [plan_type:Utf8, plan:Utf8]", "  Projection: t1.t1_id, t1.t1_name, t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "    Filter: EXISTS (<subquery>) [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "      Subquery: [t1_int:U ...

Right: ["Explain [plan_type:Utf8, plan:Utf8]", "  Filter: EXISTS (<subquery>) [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "    Subquery: [t1_int:UInt32;N]", "      Projection: t1.t1_int [t1_int:UInt32;N]", "        Filter: t1.t1_id > t1.t1_int [t1_i ...

<Click to see difference>

thread 'sql::subqueries::exists_subquery_with_same_table' panicked at 'assertion failed: `(left == right)`
  left: `["Explain [plan_type:Utf8, plan:Utf8]", "  Projection: t1.t1_id, t1.t1_name, t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "    Filter: EXISTS (<subquery>) [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "      Subquery: [t1_int:UInt32;N]", "        Projection: t1.t1_int [t1_int:UInt32;N]", "          Filter: t1.t1_id > t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "            TableScan: t1 [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "      TableScan: t1 projection=[t1_id, t1_name, t1_int] [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]"]`,
 right: `["Explain [plan_type:Utf8, plan:Utf8]", "  Filter: EXISTS (<subquery>) [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "    Subquery: [t1_int:UInt32;N]", "      Projection: t1.t1_int [t1_int:UInt32;N]", "        Filter: t1.t1_id > t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "          TableScan: t1 [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "    TableScan: t1 projection=[t1_id, t1_name, t1_int] [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]"]`: 

expected:

[
    "Explain [plan_type:Utf8, plan:Utf8]",
    "  Projection: t1.t1_id, t1.t1_name, t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "    Filter: EXISTS (<subquery>) [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "      Subquery: [t1_int:UInt32;N]",
    "        Projection: t1.t1_int [t1_int:UInt32;N]",
    "          Filter: t1.t1_id > t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "            TableScan: t1 [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "      TableScan: t1 projection=[t1_id, t1_name, t1_int] [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
]
actual:

[
    "Explain [plan_type:Utf8, plan:Utf8]",
    "  Filter: EXISTS (<subquery>) [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "    Subquery: [t1_int:UInt32;N]",
    "      Projection: t1.t1_int [t1_int:UInt32;N]",
    "        Filter: t1.t1_id > t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "          TableScan: t1 [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "    TableScan: t1 projection=[t1_id, t1_name, t1_int] [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
]

', datafusion/core/tests/sql/subqueries.rs:143:5
stack backtrace:
   0: rust_begin_unwind
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/std/src/panicking.rs:575:5
   1: core::panicking::panic_fmt
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/core/src/panicking.rs:64:14
   2: core::panicking::assert_failed_inner
   3: core::panicking::assert_failed
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/core/src/panicking.rs:211:5
   4: sql_integration::sql::subqueries::exists_subquery_with_same_table::{{closure}}
             at ./tests/sql/subqueries.rs:143:5
   5: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/core/src/future/future.rs:125:9
   6: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:525:48
   7: tokio::coop::with_budget::{{closure}}
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:102:9
   8: std::thread::local::LocalKey<T>::try_with
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/std/src/thread/local.rs:446:16
   9: std::thread::local::LocalKey<T>::with
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/std/src/thread/local.rs:422:9
  10: tokio::coop::with_budget
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:95:5
  11: tokio::coop::budget
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:72:5
  12: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:525:25
  13: tokio::runtime::scheduler::current_thread::Context::enter
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:349:19
  14: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:524:36
  15: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:595:57
  16: tokio::macros::scoped_tls::ScopedKey<T>::set
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/macros/scoped_tls.rs:61:9
  17: tokio::runtime::scheduler::current_thread::CoreGuard::enter
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:595:27
  18: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:515:19
  19: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:161:24
  20: tokio::runtime::Runtime::block_on
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/mod.rs:490:46
  21: sql_integration::sql::subqueries::exists_subquery_with_same_table
             at ./tests/sql/subqueries.rs:148:5
  22: sql_integration::sql::subqueries::exists_subquery_with_same_table::{{closure}}
             at ./tests/sql/subqueries.rs:121:47
  23: core::ops::function::FnOnce::call_once
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/core/src/ops/function.rs:250:5
  24: core::ops::function::FnOnce::call_once
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.




expected:

[
    "Explain [plan_type:Utf8, plan:Utf8]",
    "  Projection: t1.t1_id, t1.t1_name, t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "    LeftSemi Join: t1.t1_id = __correlated_sq_1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "      TableScan: t1 projection=[t1_id, t1_name, t1_int] [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "      SubqueryAlias: __correlated_sq_1 [t1_int:UInt32;N]",
    "        Projection: t1.t1_int AS t1_int [t1_int:UInt32;N]",
    "          Filter: t1.t1_id > t1.t1_int [t1_id:UInt32;N, t1_int:UInt32;N]",
    "            TableScan: t1 projection=[t1_id, t1_int] [t1_id:UInt32;N, t1_int:UInt32;N]",
]
actual:

[
    "Explain [plan_type:Utf8, plan:Utf8]",
    "  LeftSemi Join: t1.t1_id = __correlated_sq_1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "    TableScan: t1 projection=[t1_id, t1_name, t1_int] [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "    SubqueryAlias: __correlated_sq_1 [t1_int:UInt32;N]",
    "      Projection: t1.t1_int AS t1_int [t1_int:UInt32;N]",
    "        Filter: t1.t1_id > t1.t1_int [t1_id:UInt32;N, t1_int:UInt32;N]",
    "          TableScan: t1 projection=[t1_id, t1_int] [t1_id:UInt32;N, t1_int:UInt32;N]",
]


Left:  ["Explain [plan_type:Utf8, plan:Utf8]", "  Projection: t1.t1_id, t1.t1_name, t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "    LeftSemi Join: t1.t1_id = __correlated_sq_1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "   ...

Right: ["Explain [plan_type:Utf8, plan:Utf8]", "  LeftSemi Join: t1.t1_id = __correlated_sq_1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "    TableScan: t1 projection=[t1_id, t1_name, t1_int] [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N] ...

<Click to see difference>

thread 'sql::subqueries::in_subquery_with_same_table' panicked at 'assertion failed: `(left == right)`
  left: `["Explain [plan_type:Utf8, plan:Utf8]", "  Projection: t1.t1_id, t1.t1_name, t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "    LeftSemi Join: t1.t1_id = __correlated_sq_1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "      TableScan: t1 projection=[t1_id, t1_name, t1_int] [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "      SubqueryAlias: __correlated_sq_1 [t1_int:UInt32;N]", "        Projection: t1.t1_int AS t1_int [t1_int:UInt32;N]", "          Filter: t1.t1_id > t1.t1_int [t1_id:UInt32;N, t1_int:UInt32;N]", "            TableScan: t1 projection=[t1_id, t1_int] [t1_id:UInt32;N, t1_int:UInt32;N]"]`,
 right: `["Explain [plan_type:Utf8, plan:Utf8]", "  LeftSemi Join: t1.t1_id = __correlated_sq_1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "    TableScan: t1 projection=[t1_id, t1_name, t1_int] [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "    SubqueryAlias: __correlated_sq_1 [t1_int:UInt32;N]", "      Projection: t1.t1_int AS t1_int [t1_int:UInt32;N]", "        Filter: t1.t1_id > t1.t1_int [t1_id:UInt32;N, t1_int:UInt32;N]", "          TableScan: t1 projection=[t1_id, t1_int] [t1_id:UInt32;N, t1_int:UInt32;N]"]`: 

expected:

[
    "Explain [plan_type:Utf8, plan:Utf8]",
    "  Projection: t1.t1_id, t1.t1_name, t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "    LeftSemi Join: t1.t1_id = __correlated_sq_1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "      TableScan: t1 projection=[t1_id, t1_name, t1_int] [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "      SubqueryAlias: __correlated_sq_1 [t1_int:UInt32;N]",
    "        Projection: t1.t1_int AS t1_int [t1_int:UInt32;N]",
    "          Filter: t1.t1_id > t1.t1_int [t1_id:UInt32;N, t1_int:UInt32;N]",
    "            TableScan: t1 projection=[t1_id, t1_int] [t1_id:UInt32;N, t1_int:UInt32;N]",
]
actual:

[
    "Explain [plan_type:Utf8, plan:Utf8]",
    "  LeftSemi Join: t1.t1_id = __correlated_sq_1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "    TableScan: t1 projection=[t1_id, t1_name, t1_int] [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "    SubqueryAlias: __correlated_sq_1 [t1_int:UInt32;N]",
    "      Projection: t1.t1_int AS t1_int [t1_int:UInt32;N]",
    "        Filter: t1.t1_id > t1.t1_int [t1_id:UInt32;N, t1_int:UInt32;N]",
    "          TableScan: t1 projection=[t1_id, t1_int] [t1_id:UInt32;N, t1_int:UInt32;N]",
]

', datafusion/core/tests/sql/subqueries.rs:174:5
stack backtrace:
   0: rust_begin_unwind
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/std/src/panicking.rs:575:5
   1: core::panicking::panic_fmt
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/core/src/panicking.rs:64:14
   2: core::panicking::assert_failed_inner
   3: core::panicking::assert_failed
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/core/src/panicking.rs:211:5
   4: sql_integration::sql::subqueries::in_subquery_with_same_table::{{closure}}
             at ./tests/sql/subqueries.rs:174:5
   5: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/core/src/future/future.rs:125:9
   6: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:525:48
   7: tokio::coop::with_budget::{{closure}}
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:102:9
   8: std::thread::local::LocalKey<T>::try_with
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/std/src/thread/local.rs:446:16
   9: std::thread::local::LocalKey<T>::with
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/std/src/thread/local.rs:422:9
  10: tokio::coop::with_budget
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:95:5
  11: tokio::coop::budget
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:72:5
  12: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:525:25
  13: tokio::runtime::scheduler::current_thread::Context::enter
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:349:19
  14: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:524:36
  15: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:595:57
  16: tokio::macros::scoped_tls::ScopedKey<T>::set
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/macros/scoped_tls.rs:61:9
  17: tokio::runtime::scheduler::current_thread::CoreGuard::enter
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:595:27
  18: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:515:19
  19: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:161:24
  20: tokio::runtime::Runtime::block_on
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/mod.rs:490:46
  21: sql_integration::sql::subqueries::in_subquery_with_same_table
             at ./tests/sql/subqueries.rs:179:5
  22: sql_integration::sql::subqueries::in_subquery_with_same_table::{{closure}}
             at ./tests/sql/subqueries.rs:152:43
  23: core::ops::function::FnOnce::call_once
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/core/src/ops/function.rs:250:5
  24: core::ops::function::FnOnce::call_once
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Likely because tests were run on the branch, there were no conflicts, then upon merge the merged output failed the tests?

jackwener · 2023-02-25T19:57:08Z

@avantgardnerio Yes, it's a classic Bors problem. It offen occur in Apache project due to cannot use merge queue.
Due to this PR is conflict with Other merged PR

github-actions bot added core Core DataFusion crate optimizer Optimizer rules substrait labels Feb 22, 2023

jackwener mentioned this pull request Feb 22, 2023

Make inline_table_scan optimize whole plan during first optimization stage. #5364

Closed

jackwener force-pushed the EliminateProject branch from c3f133f to 6fee90c Compare February 22, 2023 12:43

jackwener mentioned this pull request Feb 23, 2023

generate new projection plan in inline_table_scan instead of discarding #5371

Merged

alamb mentioned this pull request Feb 23, 2023

reimplement push_down_projection and prune_column. #4465

Merged

jackwener force-pushed the EliminateProject branch from 6fee90c to 903416a Compare February 25, 2023 04:31

jackwener marked this pull request as ready for review February 25, 2023 04:31

github-actions bot added the sqllogictest SQL Logic Tests (.slt) label Feb 25, 2023

jackwener force-pushed the EliminateProject branch 3 times, most recently from c02de2f to 6276d4c Compare February 25, 2023 09:05

feat: add eliminate_unnecessary_projection rule

7478c36

jackwener force-pushed the EliminateProject branch from 6276d4c to 7478c36 Compare February 25, 2023 09:52

alamb reviewed Feb 25, 2023

View reviewed changes

alamb approved these changes Feb 25, 2023

View reviewed changes

jackwener merged commit 8b92b9b into apache:main Feb 25, 2023

jackwener deleted the EliminateProject branch February 25, 2023 15:41

avantgardnerio mentioned this pull request Feb 25, 2023

Empty commit to verify main is broken #5401

Closed

jackwener mentioned this pull request Feb 25, 2023

enhance: remove more projection. #5402

Merged

andygrove added the enhancement New feature or request label Mar 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: eliminate unnecessary projection. #5366

feat: eliminate unnecessary projection. #5366

jackwener commented Feb 22, 2023 •

edited

Loading

alamb left a comment

alamb Feb 25, 2023

ursabot commented Feb 25, 2023

avantgardnerio commented Feb 25, 2023

jackwener commented Feb 25, 2023 •

edited

Loading

feat: eliminate unnecessary projection. #5366

feat: eliminate unnecessary projection. #5366

Conversation

jackwener commented Feb 22, 2023 • edited Loading

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

alamb left a comment

Choose a reason for hiding this comment

alamb Feb 25, 2023

Choose a reason for hiding this comment

ursabot commented Feb 25, 2023

avantgardnerio commented Feb 25, 2023

jackwener commented Feb 25, 2023 • edited Loading

jackwener commented Feb 22, 2023 •

edited

Loading

jackwener commented Feb 25, 2023 •

edited

Loading