Roadmap: Fix flaky tests

[hcho3]

This issue is aimed at keeping track of flaky tests.

Rationale: some of existing tests tend to fail randomly, even when the pull request / branch update has nothing do to the tests. Since flaky tests bewilder community contributors and hamper developer productivity, they should be quickly addressed.

Note to maintainers: If you spot a flaky test, please report it here.

List of existing flaky tests

• R-package/tests/testthat/test_update.R: Test that model updates work.
  • Lines 32, 41 and 64 randomly fail for 32-bit Windows + MinGW target.
  • Build log (AppVeyor): https://ci.appveyor.com/project/tqchen/xgboost/build/1.0.2949/job/7i0d2lim73hhy44f#L2960, https://ci.appveyor.com/project/tqchen/xgboost/build/1.0.2954/job/l2icjm6yq4kc2k3s#L3093
• tests/python/regression_test_utilities.py: Test suite
  • Randomly throws std::bad_alloc exception due to memory shortage
  • Build log (Travis CI): https://travis-ci.org/dmlc/xgboost/jobs/427691975#L2250

[khotilov]

I could not reproduce the 1st one, but I will need to try with this specific version R-3.5.1patched later.

[hcho3]

@khotilov I think the issue only appears for 32-bit R.

[hcho3]

For tests/python/regression_test_utilities.py, we could solve it by migrating Python test to Jenkins (https://xgboost-ci.net), where we can use slave workers with more memory.

[khotilov]

I ran the test with some debug printouts many times locally and in a personal appveyor, but have never seen it fail :( I'm really curious to see the details for how it fails. How frequently would you estimate the failures were happening?

[hcho3]

@khotilov I managed to re-produce the error as follows:

1. Create a fresh Windows EC2 instance.
2. Install R and Rtools.
3. Compile XGBoost from source
4. Install XGBoost into 32-bit R
5. Run R tests inside 32-bit R

[trivialfis]

@hcho3 Another weird test, on XGBoost Travis test:
https://travis-ci.org/dmlc/xgboost/jobs/448105204

On my Travis test:
https://travis-ci.com/trivialfis/xgboost/jobs/155019807

I also have some test failures on my machine but not on Travis, including shougun test and this one #3622.

[hcho3]

Closing in favor of #4234