Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework some python tests of Parquet delta encodings #15693

Merged
merged 3 commits into from
May 8, 2024

Conversation

etseidl
Copy link
Contributor

@etseidl etseidl commented May 7, 2024

Description

test_parquet.py currently takes around 55s to run on an RTXA6000 system. A large portion of that run time is in two tests of the Parquet DELTA_LENGTH_BYTE_ARRAY and DELTA_BYTE_ARRAY encodings. These tests are parameterized with varying row counts to test certain encoding edge cases, but the final two row counts (10,000, 50,000) are unnecessarily large to provide adequate test coverage. This PR reduces the number of row counts (some were redundant) and decreases the maximum row count to 1,000. This drops the execution time to just under 26s on the same system.

This PR also corrects an oversight from #15239. DELTA_BYTE_ARRAY encoding should have been added to the tests at that time.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@etseidl etseidl requested a review from a team as a code owner May 7, 2024 19:30
@etseidl etseidl requested review from vyasr and bdice May 7, 2024 19:30
Copy link

copy-pr-bot bot commented May 7, 2024

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions bot added the Python Affects Python cuDF API. label May 7, 2024
@vuule vuule added tests Unit testing for project improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels May 7, 2024
@vuule
Copy link
Contributor

vuule commented May 7, 2024

/ok to test

@vuule
Copy link
Contributor

vuule commented May 8, 2024

/merge

@rapids-bot rapids-bot bot merged commit d29af84 into rapidsai:branch-24.06 May 8, 2024
70 checks passed
@etseidl etseidl deleted the delta_pytest_speedup branch May 8, 2024 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement Improvement / enhancement to an existing function non-breaking Non-breaking change Python Affects Python cuDF API. tests Unit testing for project
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants