Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: minimal support for dask.dataframe query planning (dask-expr) #285

Merged
merged 12 commits into from
May 6, 2024
Merged

ENH: minimal support for dask.dataframe query planning (dask-expr) #285

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
9567c21
duplicate core.py
jorisvandenbossche Mar 11, 2024
d28521f
support dask-expr
jorisvandenbossche Mar 11, 2024
80976e6
enable in tests
jorisvandenbossche Mar 11, 2024
01f0b86
Merge remote-tracking branch 'upstream/main' into dask-expr
May 4, 2024
2b9af30
fixed from_dask_dataframe geometry
May 4, 2024
4ee6a51
Fixed from_geopandas, to_dask_dataframe
May 4, 2024
720fa97
fixed test
May 4, 2024
d850239
Merge pull request #1 from TomAugspurger/dask-expr
jorisvandenbossche May 6, 2024
1000144
restore to_dask_dataframe (fix to_legacy_dataframe)
jorisvandenbossche May 6, 2024
5592911
fix doc requirements
jorisvandenbossche May 6, 2024
501f8dc
small cleanup
jorisvandenbossche May 6, 2024
8ef4966
add one build with recent but pre-expressions dask version
jorisvandenbossche May 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion dask_geopandas/expr.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ def __init__(self, expr, spatial_partitions=None):
super().__init__(expr)
self.spatial_partitions = spatial_partitions

def to_legacy_dataframe(self, optimize: bool = True, **optimize_kwargs):
def to_dask_dataframe(self):
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TomAugspurger I renamed this back to the original name, but now this is fine because dask-expr upstream renamed the to_dask_dataframe they added to to_legacy_dataframe, so there is no longer a name clash.

to_legacy_dataframe will convert to a legacy dask.dataframe collection, which is needed for certain implementations that still use the original implementation (e.g. dask still does this for parquet IO right now).

While our to_dask_dataframe is meant to convert your dask-geopandas object to a dask object (regardless of it being a legacy collection or a new expression).

(and so naming this implementation to_legacy_dataframe actually broke the parquet tests, because it is not doing what dask is expecting, i.e. it doesn't return a legacy collection, just the same object but where the partitions are pd.DataFrames instead of GeoDataFrames)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, makes sense, thanks.

"""Create a dask.dataframe object from a dask_geopandas object"""
return self.map_partitions(pd.DataFrame)

Expand Down
Loading