Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] Enable zstd integration tests for parquet and orc #5580

Closed
jbrennan333 opened this issue May 20, 2022 · 4 comments · Fixed by #5898 or #5991
Closed

[FEA] Enable zstd integration tests for parquet and orc #5580

jbrennan333 opened this issue May 20, 2022 · 4 comments · Fixed by #5898 or #5991
Assignees
@jbrennan333
Labels
test Only impacts tests

Comments

@jbrennan333
Copy link
Contributor

We do not currently test parquet with zstd codec. Once CUDF PR rapidsai/cudf#10851 is merged, we should enable zstd testing by changing this line in parquet_test.py:

-parquet_compress_options = ['none', 'uncompressed', 'snappy', 'gzip']
+parquet_compress_options = ['none', 'uncompressed', 'snappy', 'gzip', 'zstd']
@jbrennan333 jbrennan333 added feature request New feature or request ? - Needs Triage Need team to review and classify labels May 20, 2022
@jbrennan333 jbrennan333 self-assigned this May 20, 2022
@sameerz sameerz added test Only impacts tests and removed feature request New feature or request ? - Needs Triage Need team to review and classify labels May 20, 2022
@jbrennan333
Copy link
Contributor Author

Note that on spark2a, I did run into issues trying to run with Spark-3.1.2. So the extra jar comment may still apply for Spark-3.1.2. It works with Spark-3.2.1.

@jbrennan333 jbrennan333 mentioned this issue Jun 23, 2022
Merged
@jbrennan333
Copy link
Contributor Author

I put up PR #5898 for this.

@jbrennan333
Copy link
Contributor Author

This change was reverted by #5916 because it was done too early.

@jbrennan333 jbrennan333 reopened this Jul 12, 2022
@jbrennan333 jbrennan333 mentioned this issue Jul 12, 2022
Merged
@jbrennan333 jbrennan333 changed the title [FEA] Enable parquet integration test for zstd [FEA] Enable zstd integration tests for parquet and orc Jul 12, 2022
@jbrennan333
Copy link
Contributor Author

Edited description to include update to orc_test.py.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jbrennan333 jbrennan333

Labels
test Only impacts tests
Projects
None yet
Milestone
No milestone
2 participants
@sameerz @jbrennan333