-
Notifications
You must be signed in to change notification settings - Fork 242
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] Databricks parquetFilters build failure in db 8.2 runtime #3191
Comments
NEED CONFIRM: we may need to target this to 21.06 and release 21.06.2 since it's long-term support version... As we have the auto-merge setup, we would like to make fix target the elder-version branch |
I tried on databricks aws 8.2 runtime using 21.08 and 21.06.01 and both write parquet fine, I"m not sure that is the issue here, investigating. |
so the couple parquet writes I did worked but running the unit tests on 8.2 fail for both 21.10 and 21.08 and likely 21.06.01 as well. Must be that in 8.2 the ParquetFilters is only hit in certain cases, which seems odd because I think only fallback tests pass, I thought on 7.3 any parquet write hit the issue. |
So I was wrong, it does fail reading parquet on 8.2 with 21.06.01 in normal read case, I missed the failure. |
pr merged |
Describe the bug
Looks like Databricks just applied the API change to 8.2, related to #3098
311db shims layer is still using old ParquetFilters func which inherited from Spark301Shims,
we will need to make an 311db shims layer update
db does not add the patch version for their runtime, this brings us many headaches
The text was updated successfully, but these errors were encountered: