You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This PR is going to set the cuDF dictionary encoding policy for parquet to ADAPTIVE (#15570)
This is to get around an issue in nvcomp zstd #15501, where too large pages are getting created and is causing zstd to not compress larger dictionary pages.
For now we can pick ALWAYS in order to retain the current behavior for Spark. We should consider the impact of this setting for different compression, especially zstd (reading and writing).
The text was updated successfully, but these errors were encountered:
This PR is going to set the cuDF dictionary encoding policy for parquet to ADAPTIVE (#15570)
This is to get around an issue in nvcomp zstd #15501, where too large pages are getting created and is causing zstd to not compress larger dictionary pages.
For now we can pick ALWAYS in order to retain the current behavior for Spark. We should consider the impact of this setting for different compression, especially zstd (reading and writing).
The text was updated successfully, but these errors were encountered: