[BUG] ARIMA pytest fails on CUDA 11.2 #3649

dantegd · 2021-03-23T23:23:46Z

Describe the bug
Just ran the pytest suite on my workstation with the new RAPIDS 11.2 packages so this is the python side of #3406, the following tests failed:

FAILED cuml/test/test_arima.py::test_predict[False-float64-key4-data4] - AssertionError: 
FAILED cuml/test/test_arima.py::test_forecast[False-10-float64-key4-data4] - AssertionError: 
FAILED cuml/test/test_arima.py::test_intervals[0.5-10-float64-key4-data4] - AssertionError: 
FAILED cuml/test/test_arima.py::test_intervals[0.95-10-float64-key4-data4] - AssertionError: 
FAILED cuml/test/test_arima.py::test_loglikelihood[False-float64-key4-data4] - AssertionError:

In PR #3679 we saw the following test failures:

    cuml.test.test_arima.test_predict[False-float64-key3-data3]
    cuml.test.test_arima.test_forecast[False-10-float64-key3-data3]
    cuml.test.test_arima.test_intervals[0.5-10-float64-key3-data3]
    cuml.test.test_arima.test_intervals[0.95-10-float64-key3-data3]

job used a V100, Ubuntu 16.04, python 3.8: https://gpuci.gpuopenanalytics.com/job/rapidsai/job/gpuci-cuda112/job/cuml/job/prb/job/cuml-gpu-test/CUDA=11.2,GPU_LABEL=gpu-v100,OS=ubuntu16.04,PYTHON=3.8/48/

Steps/Code to reproduce bug
Build libcuml and cuml on CUDA 11.2

Expected behavior
Tests pass

Environment details (please complete the following information):

Environment location: Bare metal
Linux Distro/Architecture: Ubuntu 20.04 AMD64
GPU Model/Driver: 3080 / 460.56
CUDA: 11.2.67 / 11.2.142 (update 1) / 11.2.152 (update 2)
Method of cuDF & cuML install: libcuml++ built from source

Additional context
Full log of the failures:

_____________________________________________________________________________________ test_predict[False-float64-key4-data4] ______________________________________________________________________________________

key = (1, 2, 1, 0, 0, 0, ...), data = ARIMAData(batch_size=2, n_obs=137, dataset='population_estimate', start=100, end=150, tolerance_integration=0.05), dtype = <class 'numpy.float64'>
simple_differencing = False

    @pytest.mark.parametrize('key, data', test_data)
    @pytest.mark.parametrize('dtype', [np.float64])
    @pytest.mark.parametrize('simple_differencing', [True, False])
    def test_predict(key, data, dtype, simple_differencing):
        """Test in-sample prediction against statsmodels (with the same values
        for the model parameters)
        """
        n_obs = data.n_obs
>       _predict_common(key, data, dtype, n_obs // 2, n_obs,
                        simple_differencing=simple_differencing)

cuml/test/test_arima.py:341: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

key = (1, 2, 1, 0, 0, 0, ...), data = ARIMAData(batch_size=2, n_obs=137, dataset='population_estimate', start=100, end=150, tolerance_integration=0.05), dtype = <class 'numpy.float64'>, start = 68, end = 137
num_steps = None, level = None, simple_differencing = False

    def _predict_common(key, data, dtype, start, end, num_steps=None, level=None,
                        simple_differencing=True):
        """Utility function used by test_predict and test_forecast to avoid
        code duplication.
        """
        order, seasonal_order, intercept = extract_order(key)
    
        y, y_cudf = get_dataset(data, dtype)
    
        # Get fit reference model
        ref_fits = get_ref_fit(data, order, seasonal_order, intercept, dtype)
    
        # Create cuML model
        cuml_model = arima.ARIMA(y_cudf, order, seasonal_order,
                                 fit_intercept=intercept, output_type='numpy',
                                 simple_differencing=simple_differencing)
    
        # Feed the parameters to the cuML model
        _statsmodels_to_cuml(ref_fits, cuml_model, order, seasonal_order,
                             intercept, dtype)
    
        # Predict or forecast
        # Reference (statsmodels)
        ref_preds = np.zeros((end - start, data.batch_size))
        for i in range(data.batch_size):
            ref_preds[:, i] = ref_fits[i].get_prediction(
                start, end - 1).predicted_mean
        if level is not None:
            ref_lower = np.zeros((end - start, data.batch_size))
            ref_upper = np.zeros((end - start, data.batch_size))
            for i in range(data.batch_size):
                temp_pred = ref_fits[i].get_forecast(num_steps)
                ci = temp_pred.summary_frame(alpha=1-level)
                ref_lower[:, i] = ci["mean_ci_lower"].to_numpy()
                ref_upper[:, i] = ci["mean_ci_upper"].to_numpy()
        # cuML
        if num_steps is None:
            cuml_pred = cuml_model.predict(start, end)
        elif level is not None:
            cuml_pred, cuml_lower, cuml_upper = \
                cuml_model.forecast(num_steps, level)
        else:
            cuml_pred = cuml_model.forecast(num_steps)
    
        # Compare results
>       np.testing.assert_allclose(cuml_pred, ref_preds, rtol=0.001, atol=0.01)
E       AssertionError: 
E       Not equal to tolerance rtol=0.001, atol=0.01
E       
E       x and y nan location mismatch:
E        x: array([[nan, nan],
E              [nan, nan],
E              [nan, nan],...
E        y: array([[ 795.443049,  854.73595 ],
E              [ 794.049708,  861.889053],
E              [ 832.253897,  875.27896 ],...

cuml/test/test_arima.py:325: AssertionError
___________________________________________________________________________________ test_forecast[False-10-float64-key4-data4] ____________________________________________________________________________________

key = (1, 2, 1, 0, 0, 0, ...), data = ARIMAData(batch_size=2, n_obs=137, dataset='population_estimate', start=100, end=150, tolerance_integration=0.05), dtype = <class 'numpy.float64'>, num_steps = 10
simple_differencing = False

    @pytest.mark.parametrize('key, data', test_data)
    @pytest.mark.parametrize('dtype', [np.float64])
    @pytest.mark.parametrize('num_steps', [10])
    @pytest.mark.parametrize('simple_differencing', [True, False])
    def test_forecast(key, data, dtype, num_steps, simple_differencing):
        """Test out-of-sample forecasting against statsmodels (with the same
        values for the model parameters)
        """
        n_obs = data.n_obs
>       _predict_common(key, data, dtype, n_obs, n_obs + num_steps, num_steps,
                        simple_differencing=simple_differencing)

cuml/test/test_arima.py:354: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

key = (1, 2, 1, 0, 0, 0, ...), data = ARIMAData(batch_size=2, n_obs=137, dataset='population_estimate', start=100, end=150, tolerance_integration=0.05), dtype = <class 'numpy.float64'>, start = 137, end = 147
num_steps = 10, level = None, simple_differencing = False

    def _predict_common(key, data, dtype, start, end, num_steps=None, level=None,
                        simple_differencing=True):
        """Utility function used by test_predict and test_forecast to avoid
        code duplication.
        """
        order, seasonal_order, intercept = extract_order(key)
    
        y, y_cudf = get_dataset(data, dtype)
    
        # Get fit reference model
        ref_fits = get_ref_fit(data, order, seasonal_order, intercept, dtype)
    
        # Create cuML model
        cuml_model = arima.ARIMA(y_cudf, order, seasonal_order,
                                 fit_intercept=intercept, output_type='numpy',
                                 simple_differencing=simple_differencing)
    
        # Feed the parameters to the cuML model
        _statsmodels_to_cuml(ref_fits, cuml_model, order, seasonal_order,
                             intercept, dtype)
    
        # Predict or forecast
        # Reference (statsmodels)
        ref_preds = np.zeros((end - start, data.batch_size))
        for i in range(data.batch_size):
            ref_preds[:, i] = ref_fits[i].get_prediction(
                start, end - 1).predicted_mean
        if level is not None:
            ref_lower = np.zeros((end - start, data.batch_size))
            ref_upper = np.zeros((end - start, data.batch_size))
            for i in range(data.batch_size):
                temp_pred = ref_fits[i].get_forecast(num_steps)
                ci = temp_pred.summary_frame(alpha=1-level)
                ref_lower[:, i] = ci["mean_ci_lower"].to_numpy()
                ref_upper[:, i] = ci["mean_ci_upper"].to_numpy()
        # cuML
        if num_steps is None:
            cuml_pred = cuml_model.predict(start, end)
        elif level is not None:
            cuml_pred, cuml_lower, cuml_upper = \
                cuml_model.forecast(num_steps, level)
        else:
            cuml_pred = cuml_model.forecast(num_steps)
    
        # Compare results
>       np.testing.assert_allclose(cuml_pred, ref_preds, rtol=0.001, atol=0.01)
E       AssertionError: 
E       Not equal to tolerance rtol=0.001, atol=0.01
E       
E       x and y nan location mismatch:
E        x: array([[nan, nan],
E              [nan, nan],
E              [nan, nan],...
E        y: array([[2192.845644, 2265.784304],
E              [2212.872881, 2285.587355],
E              [2233.815809, 2306.914074],...

cuml/test/test_arima.py:325: AssertionError
____________________________________________________________________________________ test_intervals[0.5-10-float64-key4-data4] ____________________________________________________________________________________

key = (1, 2, 1, 0, 0, 0, ...), data = ARIMAData(batch_size=2, n_obs=137, dataset='population_estimate', start=100, end=150, tolerance_integration=0.05), dtype = <class 'numpy.float64'>, num_steps = 10
level = 0.5

    @pytest.mark.parametrize('key, data', test_data)
    @pytest.mark.parametrize('dtype', [np.float64])
    @pytest.mark.parametrize('num_steps', [10])
    @pytest.mark.parametrize('level', [0.5, 0.95])
    def test_intervals(key, data, dtype, num_steps, level):
        """Test forecast confidence intervals against statsmodels (with the same
        values for the model parameters)
        """
        n_obs = data.n_obs
>       _predict_common(key, data, dtype, n_obs, n_obs + num_steps, num_steps,
                        level)

cuml/test/test_arima.py:367: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

key = (1, 2, 1, 0, 0, 0, ...), data = ARIMAData(batch_size=2, n_obs=137, dataset='population_estimate', start=100, end=150, tolerance_integration=0.05), dtype = <class 'numpy.float64'>, start = 137, end = 147
num_steps = 10, level = 0.5, simple_differencing = True

    def _predict_common(key, data, dtype, start, end, num_steps=None, level=None,
                        simple_differencing=True):
        """Utility function used by test_predict and test_forecast to avoid
        code duplication.
        """
        order, seasonal_order, intercept = extract_order(key)
    
        y, y_cudf = get_dataset(data, dtype)
    
        # Get fit reference model
        ref_fits = get_ref_fit(data, order, seasonal_order, intercept, dtype)
    
        # Create cuML model
        cuml_model = arima.ARIMA(y_cudf, order, seasonal_order,
                                 fit_intercept=intercept, output_type='numpy',
                                 simple_differencing=simple_differencing)
    
        # Feed the parameters to the cuML model
        _statsmodels_to_cuml(ref_fits, cuml_model, order, seasonal_order,
                             intercept, dtype)
    
        # Predict or forecast
        # Reference (statsmodels)
        ref_preds = np.zeros((end - start, data.batch_size))
        for i in range(data.batch_size):
            ref_preds[:, i] = ref_fits[i].get_prediction(
                start, end - 1).predicted_mean
        if level is not None:
            ref_lower = np.zeros((end - start, data.batch_size))
            ref_upper = np.zeros((end - start, data.batch_size))
            for i in range(data.batch_size):
                temp_pred = ref_fits[i].get_forecast(num_steps)
                ci = temp_pred.summary_frame(alpha=1-level)
                ref_lower[:, i] = ci["mean_ci_lower"].to_numpy()
                ref_upper[:, i] = ci["mean_ci_upper"].to_numpy()
        # cuML
        if num_steps is None:
            cuml_pred = cuml_model.predict(start, end)
        elif level is not None:
            cuml_pred, cuml_lower, cuml_upper = \
                cuml_model.forecast(num_steps, level)
        else:
            cuml_pred = cuml_model.forecast(num_steps)
    
        # Compare results
>       np.testing.assert_allclose(cuml_pred, ref_preds, rtol=0.001, atol=0.01)
E       AssertionError: 
E       Not equal to tolerance rtol=0.001, atol=0.01
E       
E       x and y nan location mismatch:
E        x: array([[nan, nan],
E              [nan, nan],
E              [nan, nan],...
E        y: array([[2192.845644, 2265.784304],
E              [2212.872881, 2285.587355],
E              [2233.815809, 2306.914074],...

cuml/test/test_arima.py:325: AssertionError
___________________________________________________________________________________ test_intervals[0.95-10-float64-key4-data4] ____________________________________________________________________________________

key = (1, 2, 1, 0, 0, 0, ...), data = ARIMAData(batch_size=2, n_obs=137, dataset='population_estimate', start=100, end=150, tolerance_integration=0.05), dtype = <class 'numpy.float64'>, num_steps = 10
level = 0.95

    @pytest.mark.parametrize('key, data', test_data)
    @pytest.mark.parametrize('dtype', [np.float64])
    @pytest.mark.parametrize('num_steps', [10])
    @pytest.mark.parametrize('level', [0.5, 0.95])
    def test_intervals(key, data, dtype, num_steps, level):
        """Test forecast confidence intervals against statsmodels (with the same
        values for the model parameters)
        """
        n_obs = data.n_obs
>       _predict_common(key, data, dtype, n_obs, n_obs + num_steps, num_steps,
                        level)

cuml/test/test_arima.py:367: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

key = (1, 2, 1, 0, 0, 0, ...), data = ARIMAData(batch_size=2, n_obs=137, dataset='population_estimate', start=100, end=150, tolerance_integration=0.05), dtype = <class 'numpy.float64'>, start = 137, end = 147
num_steps = 10, level = 0.95, simple_differencing = True

    def _predict_common(key, data, dtype, start, end, num_steps=None, level=None,
                        simple_differencing=True):
        """Utility function used by test_predict and test_forecast to avoid
        code duplication.
        """
        order, seasonal_order, intercept = extract_order(key)
    
        y, y_cudf = get_dataset(data, dtype)
    
        # Get fit reference model
        ref_fits = get_ref_fit(data, order, seasonal_order, intercept, dtype)
    
        # Create cuML model
        cuml_model = arima.ARIMA(y_cudf, order, seasonal_order,
                                 fit_intercept=intercept, output_type='numpy',
                                 simple_differencing=simple_differencing)
    
        # Feed the parameters to the cuML model
        _statsmodels_to_cuml(ref_fits, cuml_model, order, seasonal_order,
                             intercept, dtype)
    
        # Predict or forecast
        # Reference (statsmodels)
        ref_preds = np.zeros((end - start, data.batch_size))
        for i in range(data.batch_size):
            ref_preds[:, i] = ref_fits[i].get_prediction(
                start, end - 1).predicted_mean
        if level is not None:
            ref_lower = np.zeros((end - start, data.batch_size))
            ref_upper = np.zeros((end - start, data.batch_size))
            for i in range(data.batch_size):
                temp_pred = ref_fits[i].get_forecast(num_steps)
                ci = temp_pred.summary_frame(alpha=1-level)
                ref_lower[:, i] = ci["mean_ci_lower"].to_numpy()
                ref_upper[:, i] = ci["mean_ci_upper"].to_numpy()
        # cuML
        if num_steps is None:
            cuml_pred = cuml_model.predict(start, end)
        elif level is not None:
            cuml_pred, cuml_lower, cuml_upper = \
                cuml_model.forecast(num_steps, level)
        else:
            cuml_pred = cuml_model.forecast(num_steps)
    
        # Compare results
>       np.testing.assert_allclose(cuml_pred, ref_preds, rtol=0.001, atol=0.01)
E       AssertionError: 
E       Not equal to tolerance rtol=0.001, atol=0.01
E       
E       x and y nan location mismatch:
E        x: array([[nan, nan],
E              [nan, nan],
E              [nan, nan],...
E        y: array([[2192.845644, 2265.784304],
E              [2212.872881, 2285.587355],
E              [2233.815809, 2306.914074],...

cuml/test/test_arima.py:325: AssertionError
__________________________________________________________________________________ test_loglikelihood[False-float64-key4-data4] ___________________________________________________________________________________

key = (1, 2, 1, 0, 0, 0, ...), data = ARIMAData(batch_size=2, n_obs=137, dataset='population_estimate', start=100, end=150, tolerance_integration=0.05), dtype = <class 'numpy.float64'>
simple_differencing = False

    @pytest.mark.parametrize('key, data', test_data)
    @pytest.mark.parametrize('dtype', [np.float64])
    @pytest.mark.parametrize('simple_differencing', [True, False])
    def test_loglikelihood(key, data, dtype, simple_differencing):
        """Test loglikelihood against statsmodels (with the same values for the
        model parameters)
        """
        order, seasonal_order, intercept = extract_order(key)
    
        y, y_cudf = get_dataset(data, dtype)
    
        # Get fit reference model
        ref_fits = get_ref_fit(data, order, seasonal_order, intercept, dtype)
    
        # Create cuML model
        cuml_model = arima.ARIMA(
            y_cudf, order, seasonal_order, fit_intercept=intercept,
            simple_differencing=simple_differencing)
    
        # Feed the parameters to the cuML model
        _statsmodels_to_cuml(ref_fits, cuml_model, order, seasonal_order,
                             intercept, dtype)
    
        # Compute loglikelihood
        cuml_llf = cuml_model.llf
        ref_llf = np.array([ref_fit.llf for ref_fit in ref_fits])
    
        # Compare results
>       np.testing.assert_allclose(cuml_llf, ref_llf, rtol=0.01, atol=0.01)
E       AssertionError: 
E       Not equal to tolerance rtol=0.01, atol=0.01
E       
E       x and y nan location mismatch:
E        x: array([nan, nan])
E        y: array([-487.688252, -428.799237])

cuml/test/test_arima.py:399: AssertionError

JohnZed · 2021-04-01T16:19:08Z

@Nyrio can you take a look?

Nyrio · 2021-04-08T16:53:49Z

After some tricky debugging, I opened a PR to fix this :)

@dantegd @JohnZed

Closes #3649 The error was in `batched_kalman_loop_kernel`: calculating `Y = a M X + b Y` when `b = 0` can still result in `NaN` if `Y` is uninitialized. One possible fix would be to initialize to zeros the component that was uninitialized. The fix I chose is to remove the unnecessary read, so no uninitialized value is accessed and we save unnecessary operations. Authors: - Louis Sugy (https://github.com/Nyrio) Approvers: - Dante Gama Dessavre (https://github.com/dantegd) URL: #3730

dantegd added bug Something isn't working ? - Needs Triage Need team to review and classify labels Mar 23, 2021

divyegala added Cython / Python Cython or Python issue tests Unit testing for project and removed ? - Needs Triage Need team to review and classify labels Mar 29, 2021

dantegd mentioned this issue Mar 31, 2021

Skip tests that fail on CUDA 11.2 #3679

Merged

JohnZed assigned Nyrio Apr 1, 2021

Nyrio mentioned this issue Apr 8, 2021

Fix NaN errors observed with ARIMA in CUDA 11.2 builds #3730

Merged

rapids-bot bot closed this as completed in #3730 Apr 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] ARIMA pytest fails on CUDA 11.2 #3649

[BUG] ARIMA pytest fails on CUDA 11.2 #3649

dantegd commented Mar 23, 2021 •

edited

Loading

JohnZed commented Apr 1, 2021

Nyrio commented Apr 8, 2021

[BUG] ARIMA pytest fails on CUDA 11.2 #3649

[BUG] ARIMA pytest fails on CUDA 11.2 #3649

Comments

dantegd commented Mar 23, 2021 • edited Loading

JohnZed commented Apr 1, 2021

Nyrio commented Apr 8, 2021

dantegd commented Mar 23, 2021 •

edited

Loading