Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Datafusion Optimizations and Integrations #447

Closed
5 tasks done
trueleo opened this issue Jun 25, 2023 · 2 comments
Closed
5 tasks done

Datafusion Optimizations and Integrations #447

trueleo opened this issue Jun 25, 2023 · 2 comments
Labels
enhancement New feature or request

Comments

@trueleo
Copy link
Contributor

trueleo commented Jun 25, 2023

Tracking issue for things to be done in order to fully integrate Datafusion with Parseable and solve some of the existing query issues.

  • Better partition pruning and query performance on order by p_timestamp DESC.
  • Add physical optimizer to remove SortMergeExec on order by queries.
  • Add physical optimizer to preserve partition order on select * from table as they are already sorted in ascending order. Sort order is now Desc and single partition preserves order for simple select query
  • Add optimization based on the fact that data is non overlapping on timestamp column
  • Integrate datafusion::catalog.
@trueleo
Copy link
Contributor Author

trueleo commented Jun 25, 2023

There is a discussion about SortMergeExec here apache/datafusion#6672

@nitisht nitisht added this to the Release 0.5.2 milestone Jun 25, 2023
@nitisht nitisht added the enhancement New feature or request label Jun 26, 2023
@nitisht nitisht removed this from the Release 0.6.2 milestone Sep 7, 2023
@nitisht
Copy link
Member

nitisht commented Dec 16, 2023

Fixed in recent commits. Closing this.

@nitisht nitisht closed this as completed Dec 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants