WIP: Patched DataFusion 45+ with unified execution plans #30
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* fix: `List` of `FixedSizeList` coercion issue in SQL * test: update sqllogictest result
…ng out of datafusion/core/datasource/listing (apache#14464) * make datafusion_catalog_listing * fix: this is a bit hacky * fixes: prettier, taplo etc * fixes: clippy * minor: permalink commit hash -> main * Tweak README * fix:prettier + wasm * prettier * Put unit tests with code --------- Co-authored-by: Andrew Lamb <[email protected]>
…ort in arrow instead (apache#14503) * refactor: replace uses of arrow_buffer and arrow_array with reexport in arrow * Remove arrow-buffer in common * Remove dependency in core * remove another ne * remove from functions-nested * remove from physical-expr * remove from physical-expr-common * Remove from physical-plan * Remove from substrait * fix datafusion-cli/Cargo.lock --------- Co-authored-by: Ian Lai <[email protected]> Co-authored-by: Andrew Lamb <[email protected]>
* Accept any uncorrelated plan when checking subquery correlation For the purpose of decorrelation, an uncorrelated plan is a unit. No verification needs to be performed on it. * Extract variable Extract variable from a long if condition involving a match. Improves readability. * Simplify control flow Handle the unhandled case returning immediately. This adds additional return point to the function, but removes subsequent if. At the point of this additional return we know why we bail out (some unhandled situation), later the None filter could be construed as a true condition. * Add more EXISTS SLT tests * Support uncorrelated EXISTS * fixup! Support uncorrelated EXISTS * fixup! Support uncorrelated EXISTS
* chore(deps): Update sqlparser to `0.54.0` * Update for API changes * Turn multi-object name into an error * Add test for unsupported join * Update datafusion/sql/src/planner.rs Co-authored-by: Jax Liu <[email protected]> --------- Co-authored-by: Jax Liu <[email protected]>
github-actions
bot
added
documentation
* Validate and unpack function arguments tersely Add a `take_function_args` helper that provides convenient unpacking of function arguments along with validation that the provided argument count matches the expected. A few functions are updated to leverage the new pattern to demonstrate its usefulness. * Add example in rust doc Co-authored-by: Andrew Lamb <[email protected]> * fix fmt * Export function utils publicly this exports only the newly added take_function_args function. all other utils members are pub(crate) * use compact format pattern Co-authored-by: Matthijs Brobbel <[email protected]> * fix example * fixup! fix example * fix license header Co-authored-by: Oleks V <[email protected]> * Name args in nvl2 and use take_function_args in execution too --------- Co-authored-by: Andrew Lamb <[email protected]> Co-authored-by: Matthijs Brobbel <[email protected]> Co-authored-by: Oleks V <[email protected]>
This commit fixes the following edge cases in the array_slice function
so that it's semantics match DuckDB:
- When begin < 0 and -begin > length, begin is clamped to the
beginning of the list.
- When step < 0 and begin = end, then the result should be a list with
the single element found at index begin/end.
Fixes apache#10548
* add fetch info to CoalescePartitionsExec * use Statistics with_fetch API on CoalescePartitionsExec * check limit_reached only if fetch is assigned
# Conflicts: # datafusion/sqllogictest/test_files/aggregate.slt # datafusion/sqllogictest/test_files/limit.slt # datafusion/sqllogictest/test_files/union.slt
alamb
force-pushed
the
alamb/test_datasource_exec
branch
from
da26b38 to
278b49b
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Patched DataFusion 45+ with unified execution plans
Based on influxdata#54
This branch contains the following PR:
DataSourceExecfor provided datasources, removeParquetExec,CsvExec, etc apache/datafusion#14224Based on commit apache@962494aThe idea is to test what impact this will have on us upstream in influxdb_iox