Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[REVIEW] Add ORC fuzz tests with varying function parameters #6571

Merged
merged 31 commits into from
Nov 12, 2020
Merged

[REVIEW] Add ORC fuzz tests with varying function parameters #6571

merged 31 commits into from
Nov 12, 2020

Conversation

galipremsagar
Copy link
Contributor

@galipremsagar galipremsagar commented Oct 21, 2020 •
edited
Loading

Partially resolves: #6001 , #6260
This PR:

  • Adds support for Orc fuzz workers(Both reader and writer)
  • Utilize pyorc to write a pandas dataframe to orc files
  • Adds varying test parameter combinations for cudf.read_orc and df.to_orc

This PR has changes already existing in #6489, extra avro related changes will vanish once #6489 is merged.

This is ready for review but dependent on #6384 for cudf._fuzz_testing.utils.compare_dataframe API

@galipremsagar
Add ability to override null_frequency
342af17
@galipremsagar
Add fuzz worker for AVRO
daaf23b
@galipremsagar
Merge remote-tracking branch 'upstream/branch-0.17' into avro_fuzz_tests
b6fd2d0
@galipremsagar
add utility to write a pandas dataframe to avro file
302b359
@galipremsagar
Merge remote-tracking branch 'upstream/branch-0.17' into avro_fuzz_tests
b7acee8
@galipremsagar
add ability to write nullable pandas dataframes into avro files
2920e7e
@galipremsagar
Merge remote-tracking branch 'upstream/branch-0.17' into avro_fuzz_tests
bdd4e9c
@galipremsagar
fix random param generator
c97642b
@galipremsagar
Update CHANGELOG.md
879adb2
@galipremsagar
Merge remote-tracking branch 'origin/avro_fuzz_tests' into orc_fuzz_t…
da4cc83 
…ests
@galipremsagar
Add orc fuzz worker and fuzz tests with varying parameters.
c535db0
@galipremsagar
Merge remote-tracking branch 'upstream/branch-0.17' into orc_fuzz_tests
e2d32eb
@galipremsagar
Fix issues in orc fuzz test
a4838a0
@galipremsagar
Add orc writer fuzz tests
6e0d0c9
@galipremsagar galipremsagar added 3 - Ready for Review Ready for review by team Python Affects Python cuDF API. 4 - Needs cuDF (Python) Reviewer labels Oct 21, 2020
@galipremsagar galipremsagar requested a review from a team as a code owner October 21, 2020 01:27
@galipremsagar galipremsagar self-assigned this Oct 21, 2020
@galipremsagar galipremsagar added the 0 - Blocked Cannot progress due to external reasons label Oct 21, 2020
@GPUtester
Copy link
Collaborator

Please update the changelog in order to start CI tests.

View the gpuCI docs here.

@codecov
Copy link

codecov bot commented Oct 21, 2020 •
edited
Loading

Codecov Report

Merging #6571 (d284eea) into branch-0.17 (a7e975b) will decrease coverage by 0.53%.
The diff coverage is 0.00%.

Impacted file tree graph

@@               Coverage Diff               @@
##           branch-0.17    #6571      +/-   ##
===============================================
- Coverage        82.25%   81.72%   -0.54%     
===============================================
  Files               94       95       +1     
  Lines            15512    15854     +342     
===============================================
+ Hits             12759    12956     +197     
- Misses            2753     2898     +145
Impacted Files Coverage Δ
python/cudf/cudf/_fuzz_testing/orc.py 0.00% <0.00%> (ø)
python/cudf/cudf/_fuzz_testing/utils.py 0.00% <0.00%> (ø)
python/cudf/cudf/core/series.py 91.25% <0.00%> (+0.06%) ⬆️
python/cudf/cudf/utils/ioutils.py 88.74% <0.00%> (+2.37%) ⬆️
python/dask_cudf/dask_cudf/core.py 77.75% <0.00%> (+3.40%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a7e975b...d284eea. Read the comment docs.

@vuule vuule requested a review from rgsl888prabhu October 31, 2020 01:04
rgsl888prabhu
rgsl888prabhu approved these changes Nov 2, 2020
View reviewed changes
Copy link
Contributor

@rgsl888prabhu rgsl888prabhu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A small question, other than that looks good.

python/cudf/cudf/_fuzz_testing/tests/fuzz_test_orc.py Show resolved Hide resolved
@vuule vuule self-requested a review November 11, 2020 07:30
kkraus14
kkraus14 reviewed Nov 11, 2020
View reviewed changes
python/cudf/cudf/_fuzz_testing/tests/fuzz_test_orc.py Outdated Show resolved Hide resolved
python/cudf/cudf/_fuzz_testing/tests/fuzz_test_orc.py Outdated Show resolved Hide resolved
python/cudf/cudf/_fuzz_testing/tests/fuzz_test_orc.py Outdated Show resolved Hide resolved
python/cudf/cudf/_fuzz_testing/utils.py Outdated Show resolved Hide resolved
python/cudf/cudf/_fuzz_testing/utils.py Outdated Show resolved Hide resolved
python/cudf/cudf/_fuzz_testing/utils.py Outdated Show resolved Hide resolved
@vuule
Copy link
Contributor

vuule commented Nov 11, 2020

rerun tests

@galipremsagar galipremsagar added 5 - Ready to Merge Testing and reviews complete, ready to merge and removed 3 - Ready for Review Ready for review by team 4 - Needs cuDF (Python) Reviewer labels Nov 11, 2020
@galipremsagar
Copy link
Contributor Author

rerun tests

1 similar comment
@galipremsagar
Copy link
Contributor Author

rerun tests

@galipremsagar galipremsagar merged commit fbf12f3 into rapidsai:branch-0.17 Nov 12, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rgsl888prabhu rgsl888prabhu rgsl888prabhu approved these changes

@vuule vuule vuule approved these changes

@kkraus14 kkraus14 kkraus14 approved these changes

Assignees

@galipremsagar galipremsagar

Labels
5 - Ready to Merge Testing and reviews complete, ready to merge Python Affects Python cuDF API.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEA] Implement cuIO fuzz tests
5 participants
@galipremsagar @GPUtester @vuule @kkraus14 @rgsl888prabhu