Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable parquet suites from Spark UT #11366

Merged
merged 58 commits into from
Sep 24, 2024
Merged

Enable parquet suites from Spark UT #11366

merged 58 commits into from
Sep 24, 2024

Conversation

Feng-Jiang28
Copy link
Contributor

@Feng-Jiang28 Feng-Jiang28 commented Aug 20, 2024 •
edited
Loading

This pr is for enabling parquet suites for UT framework. I used some Gluten test cases in the suite as the UT frame used to.
Besides that, there are more suites and Gluten doesn't cover and Spark 3.3.0 has.
Enabled suites include:

  1. RapidsParquetAvroCompatibilitySuite
  2. RapidsParquetColumnIndexSuite
  3. RapidsParquetCompressionCodecPrecedenceSuite
  4. RapidsParquetDeltaByteArrayEncodingSuite
  5. RapidsParquetDeltaEncodingInteger
  6. RapidsParquetDeltaEncodingLong
  7. RapidsParquetDeltaLengthByteArrayEncodingSuite
  8. RapidsParquetFieldIdIOSuite
  9. RapidsParquetFieldIdSchemaSuite
  10. RapidsParquetFileFormatSuite
  11. RapidsParquetInteroperabilitySuite
  12. RapidsParquetProtobufCompatibilitySuite
  13. RapidsParquetPartitionDiscoverySuite
  14. RapidsParquetQuerySuite
  15. RapidsParquetRebaseDatetimeSuite
  16. RapidsParquetSchemaPruningSuite
  17. RapidsParquetSchemaSuite
  18. RapidsParquetThriftCompatibilitySuite
  19. RapidsParquetVectorizedSuite

V1 and V2 versions of the data source API related suites were not enabled or put in the tests.

fejiang and others added 11 commits August 8, 2024 15:07
add parquet column index ut test
3eee7bb 
Signed-off-by: fejiang <[email protected]>
change
4267527 
Signed-off-by: fejiang <[email protected]>
@Feng-Jiang28
Merge remote-tracking branch 'upstream/branch-24.08' into UTEnablePar…
9f8f50f 
…quetTests
@Feng-Jiang28
Merge remote-tracking branch 'upstream/branch-24.10' into UTEnablePar…
a030eb6 
…quetTests
@Feng-Jiang28
added parquet suite
176e52b 
Signed-off-by: fejiang <[email protected]>
@Feng-Jiang28
Merge remote-tracking branch 'upstream/branch-24.10' into UTEnablePar…
ffac949 
…quetTests
@Feng-Jiang28
pom changed
5ca8717 
Signed-off-by: fejiang <[email protected]>
@Feng-Jiang28
DeltaEncoding Suite
a5778e2 
Signed-off-by: fejiang <[email protected]>
@Feng-Jiang28
enable more suites
0b54f27 
Signed-off-by: fejiang <[email protected]>
@Feng-Jiang28
remove ignored case
41d9d64 
Signed-off-by: fejiang <[email protected]>
@Feng-Jiang28
format
36fb01f 
Signed-off-by: fejiang <[email protected]>
@Feng-Jiang28 Feng-Jiang28 added ? - Needs Triage Need team to review and classify test Only impacts tests and removed ? - Needs Triage Need team to review and classify labels Aug 20, 2024
@Feng-Jiang28 Feng-Jiang28 marked this pull request as draft August 20, 2024 12:00
@Feng-Jiang28
added ignored cases
893058c 
Signed-off-by: fejiang <[email protected]>
jlowe
jlowe reviewed Aug 20, 2024
View reviewed changes
tests/pom.xml Outdated Show resolved Hide resolved
tests/pom.xml Show resolved Hide resolved
@Feng-Jiang28 Feng-Jiang28 changed the title PR for enabling parquet suites [WIP]PR for enabling parquet suites Aug 21, 2024
fejiang and others added 6 commits August 21, 2024 09:43
change to parquet hadoop version
28b00ca 
Signed-off-by: fejiang <[email protected]>
change
1e62076 
Signed-off-by: fejiang <[email protected]>
remove parquet.version
a62f883 
Signed-off-by: fejiang <[email protected]>
adding scope and classifier
dc5e855 
Signed-off-by: fejiang <[email protected]>
@Feng-Jiang28
pom remove unused
ca64027 
Signed-off-by: fejiang <[email protected]>
@Feng-Jiang28
pom chang3 2.13
77f18a0 
Signed-off-by: fejiang <[email protected]>
@Feng-Jiang28 Feng-Jiang28 mentioned this pull request Sep 6, 2024
Open
@Feng-Jiang28
enabling more suite
2cd6c3e 
Signed-off-by: fejiang <[email protected]>
@Feng-Jiang28
Copy link
Contributor Author

build

@Feng-Jiang28
Copy link
Contributor Author

build

@Feng-Jiang28
Copy link
Contributor Author

build

1 similar comment
@Feng-Jiang28
Copy link
Contributor Author

build

thirtiseven
thirtiseven reviewed Sep 18, 2024
View reviewed changes
Copy link
Collaborator

@thirtiseven thirtiseven left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are output4.parquet and .output4.parquet.crc added on purpose?

.../spark330/scala/org/apache/spark/sql/rapids/suites/RapidsParquetAvroCompatibilitySuite.scala Show resolved Hide resolved
@Feng-Jiang28 Feng-Jiang28 changed the title [WIP]PR for enabling parquet suites PR for enabling parquet suites Sep 19, 2024
@Feng-Jiang28 Feng-Jiang28 marked this pull request as ready for review September 19, 2024 02:15
@Feng-Jiang28
Copy link
Contributor Author

build

binmahone
binmahone requested changes Sep 23, 2024
View reviewed changes
scala2.13/tests/pom.xml Outdated
<dependency>
<groupId>org.apache.parquet</groupId>
<artifactId>parquet-avro</artifactId>
<version>1.12.2</version>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use variable substitution?

...c/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsParquetColumnIndexSuite.scala Outdated Show resolved Hide resolved
@Feng-Jiang28
Copy link
Contributor Author

build

1 similar comment
@Feng-Jiang28
Copy link
Contributor Author

build

@thirtiseven thirtiseven changed the title PR for enabling parquet suites Enable parquet suites from Spark UT Sep 24, 2024
thirtiseven
thirtiseven reviewed Sep 24, 2024
View reviewed changes
Copy link
Collaborator

@thirtiseven thirtiseven left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but not a pom change expert

@Feng-Jiang28 Feng-Jiang28 merged commit a34f33e into NVIDIA:branch-24.10 Sep 24, 2024
44 of 45 checks passed
This was referenced Sep 27, 2024
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@thirtiseven thirtiseven thirtiseven left review comments

@binmahone binmahone binmahone approved these changes

@GaryShen2008 GaryShen2008 GaryShen2008 approved these changes

@revans2 revans2 Awaiting requested review from revans2 revans2 is a code owner

@tgravescs tgravescs Awaiting requested review from tgravescs tgravescs is a code owner

@NvTimLiu NvTimLiu Awaiting requested review from NvTimLiu NvTimLiu is a code owner

@gerashegalov gerashegalov Awaiting requested review from gerashegalov gerashegalov is a code owner

@jlowe jlowe Awaiting requested review from jlowe

Assignees
No one assigned
Labels
test Only impacts tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@Feng-Jiang28 @jlowe @binmahone @thirtiseven @GaryShen2008