Try to push down full filter before break-up #5367

avantgardnerio · 2023-02-22T21:00:01Z

Which issue does this PR close?

Closes #5357.

Rationale for this change

Allow TableProviders with multi-column indexes to fully filter results. Presently, if a TableProvider is ordered by [col_a, col_b], and it gets a query where col_a=1 and col_b=2, then it receives two calls to supports_filter_pushdown(): one for col_a and one for col_b and it would have to return exact for col_a and Unsupported for col_b because it can filter them together, and it can filter col_a because it's first in the sort order, but it cannot filter col_b on it's own.

This change would allow the TableProvider to say "Yes, I can filter expressions like col_a = x and col_b = y.

What changes are included in this PR?

Change the TableProvider API to have a single method that accepts the entire Vec of Exprs and returns a Vec of results so the TableProvider can assess them all wholistically.

Are these changes tested?

yes.

Are there any user-facing changes?

No

avantgardnerio · 2023-02-22T21:01:04Z

As an alternative to jumping to full support of ordered indexes, here is a lighter weight PR that would still unblock me.

alamb · 2023-02-23T12:55:14Z

I have a question / comment: #5357 (comment)

yjshen

LGTM.

yjshen · 2023-02-25T04:40:09Z

datafusion/expr/src/table_source.rs

+    ) -> Result<Vec<TableProviderFilterPushDown>> {
+        filters
+            .iter()
+            .map(|f| self.supports_filter_pushdown(f))


thinkharderdev

Seems reasonable to me

datafusion/optimizer/src/push_down_filter.rs

avantgardnerio · 2023-02-25T19:44:44Z

These tests are showing up as broken in this branch, but they appear to be broken in main as well:

expected:

[
    "Explain [plan_type:Utf8, plan:Utf8]",
    "  Projection: t1.t1_id, t1.t1_name, t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "    Filter: EXISTS (<subquery>) [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "      Subquery: [t1_int:UInt32;N]",
    "        Projection: t1.t1_int [t1_int:UInt32;N]",
    "          Filter: t1.t1_id > t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "            TableScan: t1 [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "      TableScan: t1 projection=[t1_id, t1_name, t1_int] [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
]
actual:

[
    "Explain [plan_type:Utf8, plan:Utf8]",
    "  Filter: EXISTS (<subquery>) [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "    Subquery: [t1_int:UInt32;N]",
    "      Projection: t1.t1_int [t1_int:UInt32;N]",
    "        Filter: t1.t1_id > t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "          TableScan: t1 [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "    TableScan: t1 projection=[t1_id, t1_name, t1_int] [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
]


Left:  ["Explain [plan_type:Utf8, plan:Utf8]", "  Projection: t1.t1_id, t1.t1_name, t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "    Filter: EXISTS (<subquery>) [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "      Subquery: [t1_int:U ...

Right: ["Explain [plan_type:Utf8, plan:Utf8]", "  Filter: EXISTS (<subquery>) [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "    Subquery: [t1_int:UInt32;N]", "      Projection: t1.t1_int [t1_int:UInt32;N]", "        Filter: t1.t1_id > t1.t1_int [t1_i ...

<Click to see difference>

thread 'sql::subqueries::exists_subquery_with_same_table' panicked at 'assertion failed: `(left == right)`
  left: `["Explain [plan_type:Utf8, plan:Utf8]", "  Projection: t1.t1_id, t1.t1_name, t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "    Filter: EXISTS (<subquery>) [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "      Subquery: [t1_int:UInt32;N]", "        Projection: t1.t1_int [t1_int:UInt32;N]", "          Filter: t1.t1_id > t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "            TableScan: t1 [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "      TableScan: t1 projection=[t1_id, t1_name, t1_int] [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]"]`,
 right: `["Explain [plan_type:Utf8, plan:Utf8]", "  Filter: EXISTS (<subquery>) [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "    Subquery: [t1_int:UInt32;N]", "      Projection: t1.t1_int [t1_int:UInt32;N]", "        Filter: t1.t1_id > t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "          TableScan: t1 [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "    TableScan: t1 projection=[t1_id, t1_name, t1_int] [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]"]`: 

expected:

[
    "Explain [plan_type:Utf8, plan:Utf8]",
    "  Projection: t1.t1_id, t1.t1_name, t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "    Filter: EXISTS (<subquery>) [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "      Subquery: [t1_int:UInt32;N]",
    "        Projection: t1.t1_int [t1_int:UInt32;N]",
    "          Filter: t1.t1_id > t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "            TableScan: t1 [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "      TableScan: t1 projection=[t1_id, t1_name, t1_int] [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
]
actual:

[
    "Explain [plan_type:Utf8, plan:Utf8]",
    "  Filter: EXISTS (<subquery>) [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "    Subquery: [t1_int:UInt32;N]",
    "      Projection: t1.t1_int [t1_int:UInt32;N]",
    "        Filter: t1.t1_id > t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "          TableScan: t1 [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "    TableScan: t1 projection=[t1_id, t1_name, t1_int] [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
]

', datafusion/core/tests/sql/subqueries.rs:143:5
stack backtrace:
   0: rust_begin_unwind
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/std/src/panicking.rs:575:5
   1: core::panicking::panic_fmt
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/core/src/panicking.rs:64:14
   2: core::panicking::assert_failed_inner
   3: core::panicking::assert_failed
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/core/src/panicking.rs:211:5
   4: sql_integration::sql::subqueries::exists_subquery_with_same_table::{{closure}}
             at ./tests/sql/subqueries.rs:143:5
   5: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/core/src/future/future.rs:125:9
   6: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:525:48
   7: tokio::coop::with_budget::{{closure}}
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:102:9
   8: std::thread::local::LocalKey<T>::try_with
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/std/src/thread/local.rs:446:16
   9: std::thread::local::LocalKey<T>::with
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/std/src/thread/local.rs:422:9
  10: tokio::coop::with_budget
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:95:5
  11: tokio::coop::budget
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:72:5
  12: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:525:25
  13: tokio::runtime::scheduler::current_thread::Context::enter
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:349:19
  14: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:524:36
  15: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:595:57
  16: tokio::macros::scoped_tls::ScopedKey<T>::set
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/macros/scoped_tls.rs:61:9
  17: tokio::runtime::scheduler::current_thread::CoreGuard::enter
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:595:27
  18: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:515:19
  19: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:161:24
  20: tokio::runtime::Runtime::block_on
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/mod.rs:490:46
  21: sql_integration::sql::subqueries::exists_subquery_with_same_table
             at ./tests/sql/subqueries.rs:148:5
  22: sql_integration::sql::subqueries::exists_subquery_with_same_table::{{closure}}
             at ./tests/sql/subqueries.rs:121:47
  23: core::ops::function::FnOnce::call_once
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/core/src/ops/function.rs:250:5
  24: core::ops::function::FnOnce::call_once
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.




expected:

[
    "Explain [plan_type:Utf8, plan:Utf8]",
    "  Projection: t1.t1_id, t1.t1_name, t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "    LeftSemi Join: t1.t1_id = __correlated_sq_1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "      TableScan: t1 projection=[t1_id, t1_name, t1_int] [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "      SubqueryAlias: __correlated_sq_1 [t1_int:UInt32;N]",
    "        Projection: t1.t1_int AS t1_int [t1_int:UInt32;N]",
    "          Filter: t1.t1_id > t1.t1_int [t1_id:UInt32;N, t1_int:UInt32;N]",
    "            TableScan: t1 projection=[t1_id, t1_int] [t1_id:UInt32;N, t1_int:UInt32;N]",
]
actual:

[
    "Explain [plan_type:Utf8, plan:Utf8]",
    "  LeftSemi Join: t1.t1_id = __correlated_sq_1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "    TableScan: t1 projection=[t1_id, t1_name, t1_int] [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "    SubqueryAlias: __correlated_sq_1 [t1_int:UInt32;N]",
    "      Projection: t1.t1_int AS t1_int [t1_int:UInt32;N]",
    "        Filter: t1.t1_id > t1.t1_int [t1_id:UInt32;N, t1_int:UInt32;N]",
    "          TableScan: t1 projection=[t1_id, t1_int] [t1_id:UInt32;N, t1_int:UInt32;N]",
]


Left:  ["Explain [plan_type:Utf8, plan:Utf8]", "  Projection: t1.t1_id, t1.t1_name, t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "    LeftSemi Join: t1.t1_id = __correlated_sq_1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "   ...

Right: ["Explain [plan_type:Utf8, plan:Utf8]", "  LeftSemi Join: t1.t1_id = __correlated_sq_1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "    TableScan: t1 projection=[t1_id, t1_name, t1_int] [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N] ...

<Click to see difference>

thread 'sql::subqueries::in_subquery_with_same_table' panicked at 'assertion failed: `(left == right)`
  left: `["Explain [plan_type:Utf8, plan:Utf8]", "  Projection: t1.t1_id, t1.t1_name, t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "    LeftSemi Join: t1.t1_id = __correlated_sq_1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "      TableScan: t1 projection=[t1_id, t1_name, t1_int] [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "      SubqueryAlias: __correlated_sq_1 [t1_int:UInt32;N]", "        Projection: t1.t1_int AS t1_int [t1_int:UInt32;N]", "          Filter: t1.t1_id > t1.t1_int [t1_id:UInt32;N, t1_int:UInt32;N]", "            TableScan: t1 projection=[t1_id, t1_int] [t1_id:UInt32;N, t1_int:UInt32;N]"]`,
 right: `["Explain [plan_type:Utf8, plan:Utf8]", "  LeftSemi Join: t1.t1_id = __correlated_sq_1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "    TableScan: t1 projection=[t1_id, t1_name, t1_int] [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]", "    SubqueryAlias: __correlated_sq_1 [t1_int:UInt32;N]", "      Projection: t1.t1_int AS t1_int [t1_int:UInt32;N]", "        Filter: t1.t1_id > t1.t1_int [t1_id:UInt32;N, t1_int:UInt32;N]", "          TableScan: t1 projection=[t1_id, t1_int] [t1_id:UInt32;N, t1_int:UInt32;N]"]`: 

expected:

[
    "Explain [plan_type:Utf8, plan:Utf8]",
    "  Projection: t1.t1_id, t1.t1_name, t1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "    LeftSemi Join: t1.t1_id = __correlated_sq_1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "      TableScan: t1 projection=[t1_id, t1_name, t1_int] [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "      SubqueryAlias: __correlated_sq_1 [t1_int:UInt32;N]",
    "        Projection: t1.t1_int AS t1_int [t1_int:UInt32;N]",
    "          Filter: t1.t1_id > t1.t1_int [t1_id:UInt32;N, t1_int:UInt32;N]",
    "            TableScan: t1 projection=[t1_id, t1_int] [t1_id:UInt32;N, t1_int:UInt32;N]",
]
actual:

[
    "Explain [plan_type:Utf8, plan:Utf8]",
    "  LeftSemi Join: t1.t1_id = __correlated_sq_1.t1_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "    TableScan: t1 projection=[t1_id, t1_name, t1_int] [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
    "    SubqueryAlias: __correlated_sq_1 [t1_int:UInt32;N]",
    "      Projection: t1.t1_int AS t1_int [t1_int:UInt32;N]",
    "        Filter: t1.t1_id > t1.t1_int [t1_id:UInt32;N, t1_int:UInt32;N]",
    "          TableScan: t1 projection=[t1_id, t1_int] [t1_id:UInt32;N, t1_int:UInt32;N]",
]

', datafusion/core/tests/sql/subqueries.rs:174:5
stack backtrace:
   0: rust_begin_unwind
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/std/src/panicking.rs:575:5
   1: core::panicking::panic_fmt
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/core/src/panicking.rs:64:14
   2: core::panicking::assert_failed_inner
   3: core::panicking::assert_failed
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/core/src/panicking.rs:211:5
   4: sql_integration::sql::subqueries::in_subquery_with_same_table::{{closure}}
             at ./tests/sql/subqueries.rs:174:5
   5: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/core/src/future/future.rs:125:9
   6: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:525:48
   7: tokio::coop::with_budget::{{closure}}
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:102:9
   8: std::thread::local::LocalKey<T>::try_with
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/std/src/thread/local.rs:446:16
   9: std::thread::local::LocalKey<T>::with
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/std/src/thread/local.rs:422:9
  10: tokio::coop::with_budget
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:95:5
  11: tokio::coop::budget
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:72:5
  12: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:525:25
  13: tokio::runtime::scheduler::current_thread::Context::enter
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:349:19
  14: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:524:36
  15: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:595:57
  16: tokio::macros::scoped_tls::ScopedKey<T>::set
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/macros/scoped_tls.rs:61:9
  17: tokio::runtime::scheduler::current_thread::CoreGuard::enter
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:595:27
  18: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:515:19
  19: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:161:24
  20: tokio::runtime::Runtime::block_on
             at /home/bgardner/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.21.2/src/runtime/mod.rs:490:46
  21: sql_integration::sql::subqueries::in_subquery_with_same_table
             at ./tests/sql/subqueries.rs:179:5
  22: sql_integration::sql::subqueries::in_subquery_with_same_table::{{closure}}
             at ./tests/sql/subqueries.rs:152:43
  23: core::ops::function::FnOnce::call_once
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/core/src/ops/function.rs:250:5
  24: core::ops::function::FnOnce::call_once
             at /rustc/75a0be98f25a4b9de5afa0e15eb016e7f9627032/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

ursabot · 2023-02-26T02:12:57Z

Benchmark runs are scheduled for baseline = 38185ca and contender = fad360d. fad360d is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

avantgardnerio requested review from alamb and andygrove February 22, 2023 21:00

github-actions bot added the optimizer Optimizer rules label Feb 22, 2023

avantgardnerio requested a review from Dandandan February 22, 2023 21:00

avantgardnerio mentioned this pull request Feb 22, 2023

Add index interface method #5362

Closed

github-actions bot added the logical-expr Logical plan and expressions label Feb 23, 2023

avantgardnerio mentioned this pull request Feb 23, 2023

supports_filter_pushdown() receives broken-up filters #5357

Closed

avantgardnerio requested a review from thinkharderdev February 23, 2023 21:22

yjshen approved these changes Feb 25, 2023

View reviewed changes

thinkharderdev approved these changes Feb 25, 2023

View reviewed changes

alamb approved these changes Feb 25, 2023

View reviewed changes

datafusion/optimizer/src/push_down_filter.rs Outdated Show resolved Hide resolved

avantgardnerio force-pushed the bg_pushdown branch 2 times, most recently from ed4f554 to 3e34c53 Compare February 26, 2023 00:28

rebase

fb8814b

avantgardnerio force-pushed the bg_pushdown branch from 3e34c53 to fb8814b Compare February 26, 2023 00:29

avantgardnerio merged commit fad360d into apache:main Feb 26, 2023

avantgardnerio deleted the bg_pushdown branch February 26, 2023 02:05

avantgardnerio mentioned this pull request Feb 27, 2023

Push down all filters to TableProvider #5418

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Try to push down full filter before break-up #5367

Try to push down full filter before break-up #5367

avantgardnerio commented Feb 22, 2023 •

edited

Loading

avantgardnerio commented Feb 22, 2023

alamb commented Feb 23, 2023

yjshen left a comment

yjshen Feb 25, 2023

thinkharderdev left a comment

avantgardnerio commented Feb 25, 2023

ursabot commented Feb 26, 2023

Try to push down full filter before break-up #5367

Try to push down full filter before break-up #5367

Conversation

avantgardnerio commented Feb 22, 2023 • edited Loading

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

avantgardnerio commented Feb 22, 2023

alamb commented Feb 23, 2023

yjshen left a comment

Choose a reason for hiding this comment

yjshen Feb 25, 2023

Choose a reason for hiding this comment

thinkharderdev left a comment

Choose a reason for hiding this comment

avantgardnerio commented Feb 25, 2023

ursabot commented Feb 26, 2023

avantgardnerio commented Feb 22, 2023 •

edited

Loading