-
-
Notifications
You must be signed in to change notification settings - Fork 179
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Evolution #818
Evolution #818
Conversation
(NB: if there is a [_common]_metadata, will not probe other schemas, assuming that file should reflect the most recent situation. So far, we only handle additional fields, and specifying explicit dtypes, not auto casting.
@yohplala, I thought you would find this idea and implementation interesting (I am not asking for you to do anything, just for your curiosity) |
@rjzamora , this is a really useful new feature! I could use just a little help integrating with dask - the two failing tests could be changed to be simpler and pass with and without this change, but I wouldn't mind your thoughts. Secondly, doing this also revealed that row-filtering, which fastparquet supports at the ParquetFile API level (in .to_pandas() ), cannot be called by dask, which calls fastparquet.core functions directly. I'm not sure what to do about that, short of duplicating the relevant code. |
Starts to fix #817
Handles