Q28 fails in automated nightly runs

[beckernick]

This is the same error as we had in #140 that was in theory resolved by rapidsai/cuml#3152 . cc @dantegd @VibhuJawa

28                                                                                                                      [958/1807]
Encountered Exception while running query
Traceback (most recent call last):
  File "/raid/nicholasb/prod/tpcx-bb/tpcx_bb/xbb_tools/utils.py", line 280, in run_dask_cudf_query
    config=config,
  File "/raid/nicholasb/prod/tpcx-bb/tpcx_bb/xbb_tools/utils.py", line 61, in benchmark
    result = func(*args, **kwargs)
  File "queries/q28/tpcx_bb_query_28.py", line 341, in main
    client=client, train_data=train_data, test_data=test_data
  File "queries/q28/tpcx_bb_query_28.py", line 285, in post_etl_processing
    model.fit(X_train, y_train)
  File "/raid/nicholasb/miniconda3/envs/rapids-tpcxbb-automated-tests/lib/python3.7/site-packages/cuml/common/memory_utils.py", l$
ne 93, in cupy_rmm_wrapper
    return func(*args, **kwargs)
  File "/raid/nicholasb/miniconda3/envs/rapids-tpcxbb-automated-tests/lib/python3.7/site-packages/cuml/dask/naive_bayes/naive_bay$
s.py", line 190, in fit
    client=self.client)
  File "/raid/nicholasb/miniconda3/envs/rapids-tpcxbb-automated-tests/lib/python3.7/site-packages/cuml/dask/common/func.py", line
63, in reduce
    workers = [(first(who_has[m.key]), m) for m in futures]
  File "/raid/nicholasb/miniconda3/envs/rapids-tpcxbb-automated-tests/lib/python3.7/site-packages/cuml/dask/common/func.py", line
63, in <listcomp>
    workers = [(first(who_has[m.key]), m) for m in futures]
  File "/raid/nicholasb/miniconda3/envs/rapids-tpcxbb-automated-tests/lib/python3.7/site-packages/toolz/itertoolz.py", line 376, i
n first
    return next(iter(seq))
StopIteration

[beckernick]

I cannot consistently reproduce this (though others have seen it as well). There may be something subtle happening with the Naive Bayes classifier.

[VibhuJawa]

@beckernick , Could you able to fetch the logs from the workers (when you see this error again) , i suspect they might have some more context.

[beckernick]

Lost the logs from the failure in the automated nightly run, unfortunately. Could not reproduce this with 100 consecutive runs of Q28. Will be triggering a few long-running tests to see if I can grab them.

I believe I saw a CUSPARSE_STATUS_NOT_INITIALIZED in the past causing the StopIteration, which on it's own might make me wonder if there's some odd behavior going on with the CUDA runtime.

However, this query clearly succeeds repeatedly on its own. Perhaps there's some unexpected interaction occurring somewhere during the full sweep.