[TASK] Investigate CI test time slowdowns

[JohnZed]

Saw a number of times ~ 2:09 for ubuntu 18.04 tests early in 0.20 release cycle.

[lowener]

Following the method described in #3228 I ran some tests with heavy CPU loads to replicate CI conditions. This resulted in much slower tests.

My specs: 24 Core CPU, 2 Quadro RTX 8000 (48 GB)
Test conditions: 31 instances of yes > /dev/null&

Test time went from 24 mins to 48 mins

Results with no load

---

============================ slowest 100 durations =============================
20.26s call     cuml/test/dask/test_nearest_neighbors.py::test_batch_size[100-10-10-1000]
17.45s call     cuml/test/test_arima.py::test_integration[float64-key6-data6]
17.18s call     cuml/test/test_umap.py::test_umap_fit_transform_trust[blobs-categorical]
10.77s call     cuml/test/test_arima.py::test_integration[float64-key7-data7]
10.26s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_and_score[dataset0-parameters1-dask_array]
10.20s call     cuml/test/dask/test_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters1-dask_array]
8.50s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_and_score[dataset0-parameters1-dask_cudf]
8.49s call     cuml/test/dask/test_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters1-dask_cudf]
8.11s call     cuml/test/test_random_forest.py::test_concat_memory_leak[large_clf0-regression]
8.05s call     cuml/test/test_random_forest.py::test_concat_memory_leak[large_clf0-classification]
7.80s call     cuml/test/test_benchmark.py::test_real_algos_runner[UMAP-Supervised]
7.01s call     cuml/test/test_umap.py::test_umap_fit_transform_trust[blobs-euclidean]
6.03s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[False-10-5-10-5-30-1000.0]
6.01s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[False-5-5-10-5-10-1000.0]
6.00s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[False-10-5-10-5-10-1000.0]
5.94s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[True-5-5-10-5-10-1000.0]
5.86s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[True-10-5-10-5-30-1000.0]
5.82s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[False-5-5-10-5-30-1000.0]
5.81s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[True-10-5-10-5-10-1000.0]
5.78s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[True-5-5-10-5-30-1000.0]
5.76s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[True-5-5-10-5-30-100]
5.73s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[True-10-5-10-5-30-100]
5.65s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[True-10-5-10-5-10-100]
5.63s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[False-5-5-10-5-30-100]
5.62s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[False-10-5-10-5-10-100]
5.56s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[True-5-5-10-5-10-100]
5.55s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[False-10-5-10-5-30-100]
5.50s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[False-5-5-10-5-10-100]
5.38s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_proba[dataset0-parameters1-dask_array]
5.32s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_and_score[dataset0-parameters0-dask_array]
5.29s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_proba[dataset0-parameters1-dask_cudf]
5.20s call     cuml/test/dask/test_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters0-dask_array]
5.13s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_and_score[dataset0-parameters2-dask_array]
5.10s call     cuml/test/dask/test_pca.py::test_pca_fit[dataframe-67-20-1000]
5.05s teardown cuml/test/dask/test_coordinate_descent.py::test_elastic_net_default[64-column_info2-500000-float64]
5.05s teardown cuml/test/dask/test_datasets.py::test_make_classification[float64-F-23-1234-20-4-4-False-100-1000]
4.90s call     cuml/test/test_thirdparty.py::test_min_max_axis[cupy-csr-False-None]
4.88s call     cuml/test/dask/test_label_binarizer.py::test_basic_functions[True-labels0]
4.87s call     cuml/test/test_random_forest.py::test_rf_regression_float64[large_reg0-datatype0]
4.82s call     cuml/test/dask/test_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters2-dask_cudf]
4.81s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_and_score[dataset0-parameters2-dask_cudf]
4.66s call     cuml/test/dask/test_umap.py::test_umap_mnmg[10-digits-True-0.4-100-2]
4.58s teardown cuml/test/dask/test_base.py::test_getattr
4.49s teardown cuml/test/dask/test_random_forest.py::test_rf_broadcast[False-False-regression]
4.43s call     cuml/test/test_api.py::test_fit_function[IncrementalPCA]
4.36s call     cuml/test/dask/test_coordinate_descent.py::test_lasso[True-4-column_info0-50-cyclic-0.001-float32]
4.20s call     cuml/test/dask/test_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters0-dask_cudf]
4.13s call     cuml/test/dask/test_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters2-dask_array]
4.12s call     cuml/test/dask/test_tsvd.py::test_pca_fit[dataframe-data_info0]
4.07s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_and_score[dataset0-parameters0-dask_cudf]
4.01s call     cuml/test/dask/test_tfidf.py::test_tfidf_transformer[True-True-True-l1-base_case]
3.94s call     cuml/test/dask/test_one_hot_encoder.py::test_onehot_vs_skonehot
3.91s call     cuml/test/dask/test_datasets.py::test_make_regression[True-F-4-True-True-1.0-0.5-None--4.0-1-7-100-1000]
3.72s call     cuml/test/test_random_forest.py::test_rf_host_memory_leak[large_clf0-classification]
3.67s call     cuml/test/test_pickle.py::test_regressor_pickle[False-data_size0-MBSGDClassifier-float32]
3.50s call     cuml/test/test_pickle.py::test_regressor_pickle[True-data_size0-MBSGDClassifier-float32]
3.36s call     cuml/test/dask/test_coordinate_descent.py::test_elastic_net[True-16-column_info0-500-random-0.5-float32]
3.32s call     cuml/test/test_pickle.py::test_regressor_pickle[False-data_size0-MBSGDClassifier-float64]
3.31s call     cuml/test/dask/test_coordinate_descent.py::test_elastic_net[True-16-column_info0-500-cyclic-0.5-float32]
3.31s call     cuml/test/test_pickle.py::test_regressor_pickle[True-data_size0-MBSGDClassifier-float64]
3.26s call     cuml/test/stemmer_tests/test_stemmer.py::test_same_results
3.15s call     cuml/test/dask/test_coordinate_descent.py::test_elastic_net[False-16-column_info0-500-random-0.5-float32]
3.09s call     cuml/test/dask/test_naive_bayes.py::test_score
3.03s call     cuml/test/dask/test_coordinate_descent.py::test_elastic_net[False-16-column_info0-500-cyclic-0.5-float32]
3.01s call     cuml/test/explainer/test_explainer_permutation_shap.py::test_regression_datasets[KNeighborsRegressor]
2.99s call     cuml/test/test_allocator.py::test_naive_bayes
2.98s call     cuml/test/dask/test_linear_regression.py::test_ols[True-float32-False-False-2-20-100000.0]
2.93s call     cuml/test/dask/test_ridge_regression.py::test_ridge[True-float32-False-False-2-10-10000.0]
2.87s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[True-5-1-10-5-10-100]
2.86s call     cuml/test/dask/test_label_encoder.py::test_labelencoder_fit_transform[5-10]
2.82s call     cuml/test/test_random_forest.py::test_rf_regression_float64[large_reg0-datatype1]
2.82s call     cuml/test/test_arima.py::test_integration[float64-key5-data5]
2.80s setup    cuml/test/dask/test_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters0-dask_array]
2.79s call     cuml/test/dask/test_naive_bayes.py::test_basic_fit_predict
2.76s call     cuml/test/dask/test_base.py::test_getattr
2.75s call     cuml/test/dask/test_base.py::test_get_combined_model[True-data_size0-LinearRegression-float32]
2.68s call     cuml/test/test_tsne.py::test_tsne_knn_graph_used[barnes_hut-sklearn-dataset0]
2.67s call     cuml/test/test_tsne.py::test_tsne_knn_graph_used[barnes_hut-cuml-dataset0]
2.61s call     cuml/test/dask/test_random_forest.py::test_rf_regression_dask_cpu[5]
2.60s call     cuml/test/dask/test_random_forest.py::test_rf_classification_multi_class[3]
2.35s call     cuml/test/dask/test_random_forest.py::test_rf_classification_dask_fil_predict_proba[5]
2.34s call     cuml/test/test_tsne.py::test_tsne_knn_graph_used[fft-cuml-dataset0]
2.32s call     cuml/test/test_umap.py::test_umap_fit_transform_trust[iris-categorical]
2.27s call     cuml/test/test_umap.py::test_umap_fit_transform_trust[digits-categorical]
2.26s call     cuml/test/dask/test_metrics.py::test_confusion_matrix[auto]
2.22s call     cuml/test/test_umap.py::test_umap_fit_transform_trust[digits-euclidean]
2.15s call     cuml/test/dask/test_kmeans.py::test_end_to_end[dataframe-True-None-5-10-1000.0]
2.14s call     cuml/test/test_tsne.py::test_tsne_knn_graph_used[fft-sklearn-dataset0]
2.11s call     cuml/test/test_benchmark.py::test_speedup_runner
2.10s call     cuml/test/test_pca.py::test_sparse_pca_inputs[False-False-True-5000-8000]
2.09s call     cuml/test/test_pca.py::test_sparse_pca_inputs[False-False-False-5000-8000]
2.08s setup    cuml/test/dask/test_base.py::test_get_combined_model[True-data_size0-LinearRegression-float32]
2.06s setup    cuml/test/test_adapters.py::test_csr_mean_variance_axis0[cupy-csr]
2.04s call     cuml/test/dask/test_dask_arr_utils.py::test_to_sparse_dask_array[10-1000-dask_dataframe]
2.04s call     cuml/test/test_pca.py::test_sparse_pca_inputs[True-False-False-5000-8000]
2.03s call     cuml/test/test_pca.py::test_sparse_pca_inputs[True-False-True-5000-8000]
2.03s call     cuml/test/test_pca.py::test_sparse_pca_inputs[True-False-True-5000-4000]
2.02s call     cuml/test/dask/test_dask_arr_utils.py::test_to_sparse_dask_array[10-1000-dask_array]
1.96s call     cuml/test/test_umap.py::test_umap_fit_transform_score[10-500]
1.94s call     cuml/test/dask/test_random_forest.py::test_rf_regression_dask_fil[5-float32]
= 20321 passed, 7424 skipped, 635 xfailed, 57 xpassed, 67788 warnings in 1483.71s (0:24:43) =

---

Results with yes load

---

============================ slowest 100 durations =============================
137.90s call     cuml/test/test_arima.py::test_integration[float64-key6-data6]
54.54s call     cuml/test/test_fil.py::test_lightgbm[5]
47.71s call     cuml/test/dask/test_nearest_neighbors.py::test_batch_size[100-10-10-1000]
39.02s call     cuml/test/test_umap.py::test_umap_fit_transform_trust[blobs-categorical]
30.64s call     cuml/test/test_incremental_pca.py::test_fit[500-2500-9-False-0.07-csr-50-True]
25.90s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_and_score[dataset0-parameters1-dask_array]
25.63s call     cuml/test/test_arima.py::test_integration[float64-key7-data7]
23.43s call     cuml/test/dask/test_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters1-dask_array]
20.29s call     cuml/test/test_umap.py::test_umap_fit_transform_trust[blobs-euclidean]
19.53s call     cuml/test/test_benchmark.py::test_real_algos_runner[UMAP-Supervised]
16.77s call     cuml/test/test_nearest_neighbors.py::test_ann_distances_metrics[l2-ivfpq]
16.46s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_and_score[dataset0-parameters1-dask_cudf]
16.21s call     cuml/test/dask/test_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters1-dask_cudf]
15.60s call     cuml/test/test_nearest_neighbors.py::test_ann_distances_metrics[euclidean-ivfpq]
15.33s call     cuml/test/test_nearest_neighbors.py::test_ann_distances_metrics[cosine-ivfpq]
15.00s call     cuml/test/test_nearest_neighbors.py::test_ann_distances_metrics[sqeuclidean-ivfpq]
14.36s call     cuml/test/test_nearest_neighbors.py::test_ann_distances_metrics[correlation-ivfpq]
14.21s call     cuml/test/test_random_forest.py::test_concat_memory_leak[large_clf0-regression]
13.99s call     cuml/test/test_random_forest.py::test_concat_memory_leak[large_clf0-classification]
13.84s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[False-10-5-10-5-30-1000.0]
13.74s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[True-10-5-10-5-30-1000.0]
13.40s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[False-5-5-10-5-30-1000.0]
13.18s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[False-5-5-10-5-10-1000.0]
13.07s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[True-5-5-10-5-30-1000.0]
12.82s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[True-5-5-10-5-10-1000.0]
12.81s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[False-10-5-10-5-10-1000.0]
12.72s call     cuml/test/test_umap.py::test_umap_fit_transform_trust[digits-euclidean]
12.54s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[False-5-5-10-5-30-100]
12.49s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[True-10-5-10-5-10-1000.0]
12.44s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[True-10-5-10-5-30-100]
12.27s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[False-10-5-10-5-30-100]
12.16s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[True-5-5-10-5-10-100]
12.08s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_proba[dataset0-parameters1-dask_array]
12.03s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[False-10-5-10-5-10-100]
11.87s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[True-10-5-10-5-10-100]
11.84s call     cuml/test/test_arima.py::test_integration[float64-key5-data5]
11.74s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[False-5-5-10-5-10-100]
11.68s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[True-5-5-10-5-30-100]
11.27s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_and_score[dataset0-parameters2-dask_array]
10.57s call     cuml/test/test_fil.py::test_lightgbm[2]
10.47s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_and_score[dataset0-parameters0-dask_array]
10.41s call     cuml/test/dask/test_coordinate_descent.py::test_lasso[True-4-column_info0-50-cyclic-0.001-float32]
10.37s call     cuml/test/dask/test_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters0-dask_array]
10.22s call     cuml/test/test_umap.py::test_umap_fit_transform_trust[digits-categorical]
9.79s call     cuml/test/test_random_forest.py::test_rf_regression_float64[large_reg0-datatype0]
9.70s call     cuml/test/dask/test_coordinate_descent.py::test_elastic_net[False-16-column_info0-500-random-0.5-float32]
9.11s call     cuml/test/dask/test_umap.py::test_umap_mnmg[10-digits-True-0.4-100-2]
9.05s call     cuml/test/dask/test_coordinate_descent.py::test_elastic_net[True-16-column_info0-500-cyclic-0.5-float32]
8.97s call     cuml/test/dask/test_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters2-dask_cudf]
8.73s call     cuml/test/dask/test_coordinate_descent.py::test_elastic_net[False-16-column_info0-500-cyclic-0.5-float32]
8.63s call     cuml/test/dask/test_pca.py::test_pca_fit[dataframe-67-20-1000]
8.61s call     cuml/test/test_random_forest.py::test_rf_classification_seed[small_clf0-float32]
8.59s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_proba[dataset0-parameters1-dask_cudf]
8.53s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_and_score[dataset0-parameters2-dask_cudf]
8.51s call     cuml/test/dask/test_coordinate_descent.py::test_elastic_net[True-16-column_info0-500-random-0.5-float32]
8.18s call     cuml/test/test_pickle.py::test_regressor_pickle[True-data_size0-MBSGDClassifier-float32]
7.89s call     cuml/test/test_pickle.py::test_regressor_pickle[False-data_size0-MBSGDClassifier-float64]
7.50s call     cuml/test/test_pickle.py::test_regressor_pickle[False-data_size0-MBSGDClassifier-float32]
7.45s call     cuml/test/dask/test_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters2-dask_array]
7.26s call     cuml/test/test_incremental_pca.py::test_fit[500-250-14-True-0.07-csr-1-True]
7.23s call     cuml/test/dask/test_tsvd.py::test_pca_fit[dataframe-data_info0]
7.19s call     cuml/test/dask/test_label_binarizer.py::test_basic_functions[True-labels0]
7.14s call     cuml/test/test_pickle.py::test_regressor_pickle[True-data_size0-MBSGDClassifier-float64]
6.90s call     cuml/test/dask/test_tfidf.py::test_tfidf_transformer[True-True-True-l1-base_case]
6.89s call     cuml/test/dask/test_one_hot_encoder.py::test_onehot_vs_skonehot
6.87s setup    cuml/test/dask/test_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters0-dask_array]
6.55s call     cuml/test/test_random_forest.py::test_rf_host_memory_leak[large_clf0-classification]
6.49s call     cuml/test/dask/test_linear_regression.py::test_ols[True-float32-False-False-2-20-100000.0]
6.43s call     cuml/test/dask/test_ridge_regression.py::test_ridge[True-float32-False-False-2-10-10000.0]
6.28s call     cuml/test/dask/test_naive_bayes.py::test_basic_fit_predict
6.26s call     cuml/test/dask/test_nearest_neighbors.py::test_compare_skl[True-5-1-10-5-10-100]
6.21s call     cuml/test/dask/test_label_encoder.py::test_labelencoder_fit_transform[5-10]
5.93s call     cuml/test/dask/test_kneighbors_classifier.py::test_predict_and_score[dataset0-parameters0-dask_cudf]
5.92s call     cuml/test/dask/test_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters0-dask_cudf]
5.55s call     cuml/test/explainer/test_explainer_permutation_shap.py::test_regression_datasets[KNeighborsRegressor]
5.45s call     cuml/test/test_random_forest.py::test_rf_regression_float64[large_reg0-datatype1]
5.32s call     cuml/test/test_umap.py::test_umap_fit_transform_trust[iris-categorical]
5.24s call     cuml/test/dask/test_base.py::test_get_combined_model[True-data_size0-LinearRegression-float32]
5.24s setup    cuml/test/test_naive_bayes.py::test_basic_fit_predict_sparse[int32-float32]
5.22s call     cuml/test/dask/test_base.py::test_getattr
5.19s call     cuml/test/dask/test_random_forest.py::test_rf_regression_dask_cpu[5]
5.10s call     cuml/test/dask/test_naive_bayes.py::test_score
5.06s teardown cuml/test/dask/test_random_forest.py::test_rf_broadcast[False-False-regression]
5.05s teardown cuml/test/dask/test_datasets.py::test_make_classification[float64-F-23-1234-20-4-4-False-100-1000]
5.04s teardown cuml/test/dask/test_coordinate_descent.py::test_elastic_net_default[64-column_info2-500000-float64]
5.02s call     cuml/test/dask/test_random_forest.py::test_rf_classification_multi_class[3]
5.01s teardown cuml/test/dask/test_base.py::test_getattr
5.01s call     cuml/test/test_trustworthiness.py::test_trustworthiness[8-100-2-500-dataframe]
4.95s call     cuml/test/test_pca.py::test_pca_defaults[False-300-200]
4.92s call     cuml/test/test_trustworthiness.py::test_trustworthiness[8-100-512-500-dataframe]
4.91s call     cuml/test/test_umap.py::test_umap_fit_transform_score_default[euclidean]
4.87s call     cuml/test/dask/test_dbscan.py::test_dbscan_propagation[int321-float32]
4.86s setup    cuml/test/dask/test_linear_regression.py::test_ols[True-float32-False-False-2-20-100000.0]
4.84s call     cuml/test/dask/test_coordinate_descent.py::test_lasso[False-4-column_info0-50-cyclic-0.001-float32]
4.79s call     cuml/test/test_trustworthiness.py::test_trustworthiness[8-10-512-500-dataframe]
4.75s setup    cuml/test/test_allocator.py::test_naive_bayes
4.64s call     cuml/test/test_trustworthiness.py::test_trustworthiness[2-100-2-500-ndarray]
4.55s call     cuml/test/test_umap.py::test_umap_fit_transform_score_default[categorical]
4.53s call     cuml/test/test_umap.py::test_umap_fit_transform_score[10-500]
4.50s call     cuml/test/stemmer_tests/test_stemmer.py::test_same_results
= 20321 passed, 7424 skipped, 635 xfailed, 57 xpassed, 67788 warnings in 2876.47s (0:47:56) =