Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate column selection/projection on dask-sql queries #236

Open
ayushdg opened this issue Feb 9, 2022 · 2 comments
Open

Investigate column selection/projection on dask-sql queries #236

ayushdg opened this issue Feb 9, 2022 · 2 comments

Comments

@ayushdg
Copy link
Member

ayushdg commented Feb 9, 2022

One Eg: In the dask-cudf query we drop columns right after a merge: https://github.com/rapidsai/gpu-bdb/blob/main/gpu_bdb/queries/q05/gpu_bdb_query_05.py#L57

The dask-sql Version does the full sql query but investigation needs to be done on whether unneeded columns are carried to the of the query and then dropped: https://github.com/rapidsai/gpu-bdb/blob/main/gpu_bdb/queries/q05/gpu_bdb_query_05_dask_sql.py#L34-L70

@randerzander
Copy link
Contributor

q02 is particularly memory heavy and specifies a subset of columns along with a WHERE clause. The intent is to minimize the size of partitions which will be shuffled, so reading the extra columns is likely very expensive.

@VibhuJawa
Copy link
Member

VibhuJawa commented Feb 10, 2022

Related dask issue: dask/dask#5809

Which has a example here: dask/dask#5809 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants