[REVIEW] Fix codecov in CI

[galipremsagar]

This PR fixes codecov reports generation when used with pytest-xdist.

[codecov]

Codecov Report

Merging #10347 (84ae7a4) into branch-22.04 (a7d88cd) will increase coverage by 75.71%.
The diff coverage is n/a.

❗ Current head 84ae7a4 differs from pull request most recent head cc78c48. Consider uploading reports for the commit cc78c48 to get more accurate results

@@                Coverage Diff                @@
##           branch-22.04   #10347       +/-   ##
=================================================
+ Coverage         10.42%   86.13%   +75.71%     
=================================================
  Files               119      139       +20     
  Lines             20603    22433     +1830     
=================================================
+ Hits               2148    19323    +17175     
+ Misses            18455     3110    -15345     

Impacted Files	Coverage Δ
...ython/custreamz/custreamz/tests/test_dataframes.py	99.39% <0.00%> (-0.01%)	⬇️
python/custreamz/custreamz/_version.py	0.00% <0.00%> (ø)
python/dask_cudf/dask_cudf/_version.py	0.00% <0.00%> (ø)
...ython/dask_cudf/dask_cudf/io/tests/test_parquet.py	100.00% <0.00%> (ø)
python/cudf/cudf/core/udf/pipeline.py
...thon/dask_cudf/dask_cudf/tests/test_distributed.py	86.79% <0.00%> (ø)
python/dask_cudf/dask_cudf/tests/test_sort.py	100.00% <0.00%> (ø)
python/cudf/cudf/core/udf/utils.py	98.63% <0.00%> (ø)
python/dask_cudf/dask_cudf/tests/test_onehot.py	100.00% <0.00%> (ø)
...ython/dask_cudf/dask_cudf/tests/test_delayed_io.py	99.03% <0.00%> (ø)
... and 105 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e610108...cc78c48. Read the comment docs.

[vyasr]

This looks great to me! It may also be nice at some point to enable the cython.Coverage plugin so that we can get coverage results for our Cython code, but the pending switch to scikit-build will bork that anyway so let's not bother with enabling coverage until we can improve scikit-build/coverage compatibility.

[vyasr]

Yes, coverage.py is extremely sensitive to file paths unfortunately.

[bdice]

Are any changes needed to measure coverage for dask-cudf and custreamz, or are those already working? Let's try to align coverage for all the Python package tests in this PR if possible.

[bdice]

Do we really want to omit our tests? Unused functions in our tests should be flagged as "not covered."

[galipremsagar]

I don't know why it was omitted previously, but I'll enable it.

[galipremsagar]

Are any changes needed to measure coverage for dask-cudf and custreamz, or are those already working?

Yea, they seem to be already working.

[bdice]

For this PR, I see https://github.com/rapidsai/cudf/blob/branch-22.04/python/dask_cudf/.coveragerc is skipping coverage for dask_cudf tests. I would recommend the same change we made for cudf, don't ignore coverage in test files.

For a follow-up, it looks like cudf/_fuzz_testing is not covered (not executed?) in our CI. Should we be running the fuzz testing code (very briefly?) to ensure it works and gives us a coverage measurement during our CI runs? I am not familiar with this part of cudf so feel free to ignore if I'm missing something here (e.g. if the code is not stable or can't be executed on CI).

[galipremsagar]

For this PR, I see https://github.com/rapidsai/cudf/blob/branch-22.04/python/dask_cudf/.coveragerc is skipping coverage for dask_cudf tests. I would recommend the same change we made for cudf, don't ignore coverage in test files.

Made the changes.

For a follow-up, it looks like cudf/_fuzz_testing is not covered (not executed?) in our CI. Should we be running the fuzz testing code (very briefly?) to ensure it works and gives us a coverage measurement during our CI runs? I am not familiar with this part of cudf so feel free to ignore if I'm missing something here (e.g. if the code is not stable or can't be executed on CI).

Those are very long-running tests as they target at the machine limits dataset sizes. Hence they are not intended to be run per PR. So we are planning on running them on benchmark machines soon.

[bdice]

Those are very long-running tests as they target at the machine limits dataset sizes. Hence they are not intended to be run per PR. So we are planning on running them on benchmark machines soon.

Right, I know fuzz testing is typically large-scale. I was wondering if there are options to use "toy" dataset sizes that can run quickly, to permit CI testing that ensures the code works and has reasonable coverage.

edit: A related problem is that we don't build or run libcudf benchmarks in CI, and sometimes PRs are merged that introduce problems in the benchmarks because we don't build them in CI (even if they didn't execute, proving they compile would be beneficial). Being able to run "CI-friendly" versions of long-running code could be helpful. Regardless, it's out of scope for this PR. Very nice job on this. 👍

[galipremsagar]

Those are very long-running tests as they target at the machine limits dataset sizes. Hence they are not intended to be run per PR. So we are planning on running them on benchmark machines soon.

Right, I know fuzz testing is typically large-scale. I was wondering if there are options to use "toy" dataset sizes that can run quickly, to permit CI testing that ensures the code works and has reasonable coverage.

Yeah, I think that is reasonable and possible, I'll try this in a follow-up PR.

[galipremsagar]

@gpucibot merge

[quasiben]

@gpucibot merge