[REVIEW] Series covariance and Pearson correlation

[beckernick]

Summary of Changes

• Adds Series-level covariance and Pearson correlation as a stopgap until they are directly supported via libcuDF
  - Adds DataFrame-level covariance and Pearson correlation via CuPy.

A future libcuDF-only version will likely be able to better support DataFrame level covariance and correlation natively. Currently, in Python, we can't do binary ops between DataFrames and Series (#2166) , nor can we do do efficient transposes. We could work around the binary op issue, but we need to transpose to leverage the matrix form of the covariance formula.

As a result, we'd need to independently calculate each covariance value. Casting nans to nulls and dropping nulls is necessary for the math, but adds a few milliseconds per cov/corr call. However, since just the covariance math itself takes 3-5 ms, we would get a significant quadratic explosion if ncols > 10-30 no matter what if we don't use the matrix formula.

Instead, we can use CuPy for now.

[beckernick]

Dask versions of any corr/cov methods will be blocked until we:

• Create an astype DataFrame method ([FEA] astype as a dataframe method #2195 ) and support the copy parameter in it
• Create a Series method for notnull. This is an alias of notna, so it should be trivial.
• Finish a df.values implementation ([FEA] Add a .values property to convert to a GPU array #1824 )

[beckernick]

This is now ready for review

[kkraus14]

@beckernick approved, suggest you open an issue for a libcudf implementation if you haven't already

[beckernick]

Will update the existing issue to reflect libcuDF implementation when we merge this. Nevermind, we didn't make an initial issue that fully covers this. Will open one when this merges and cross-reference.

This PR addresses #1267

[kkraus14]

____________ test_array_func_missing_cudf_series[<lambda>0-np_ar0] _____________

np_ar = array([0.85203364, 0.37148003, 0.24277552, 0.6316116 , 0.8227346 ,
       0.13154083, 0.59179169, 0.68867617, 0.682033...48, 0.30085795, 0.55301161, 0.92713472, 0.34632781,
       0.31003306, 0.79814686, 0.76295427, 0.04999537, 0.89813435])
func = <function <lambda> at 0x7ff21fcabae8>

    @pytest.mark.skipif(missing_arrfunc_cond, reason=missing_arrfunc_reason)
    @pytest.mark.parametrize("np_ar", [np.random.random(100)])
    @pytest.mark.parametrize(
        "func",
        [
            lambda x: np.cov(x, x),
            lambda x: np.dot(x, x),
            lambda x: np.linalg.norm(x),
        ],
    )
    def test_array_func_missing_cudf_series(np_ar, func):
        cudf_ser = cudf.Series(np_ar)
        with pytest.raises(TypeError):
>           func(cudf_ser)
E           Failed: DID NOT RAISE <class 'TypeError'>

cudf/tests/test_array_function.py:53: Failed

[beckernick]

I can't reproduce this test failure

[beckernick]

rerun tests

[kkraus14]

@beckernick You need to set the environment variable: export NUMPY_EXPERIMENTAL_ARRAY_FUNCTION=1, I'm guessing it's because you're implementing cov that it's working for numpy __array_function__ np.cov

[beckernick]

Yep, you're right. Will update the test

[codecov]

Codecov Report

Merging #2719 into branch-0.10 will increase coverage by 0.05%.
The diff coverage is 100%.

@@               Coverage Diff               @@
##           branch-0.10    #2719      +/-   ##
===============================================
+ Coverage        86.75%   86.81%   +0.05%     
===============================================
  Files               49       49              
  Lines             9055     9081      +26     
===============================================
+ Hits              7856     7884      +28     
+ Misses            1199     1197       -2

Impacted Files	Coverage Δ
python/cudf/cudf/core/series.py	93.7% <100%> (+0.35%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a902d7c...bf591b8. Read the comment docs.

[kkraus14]

rerun tests

[kkraus14]

rerun tests