[RFC] Improving continuous integration

[hcho3]

Now that we have two sponsors funding the continuous integration (CI) infrastructure (https://xgboost-ci.net), we should discuss ways to improve it.

De-couple builds from tests

Currently, we have a single Jenkins stage where XGBoost is both built and run. We should split this stage into two, one for builds and another for tests. The benefits of de-coupling compilation from test runs are:

• Eliminate redundant compilation: GPU code is quite slow in compilation, and right now we compile XGBoost many times over. Instead, compile XGBoost only once for each CUDA target.
• Test cross-version CUDA support, e.g. Test whether XGBoost package built with CUDA 8.x also runs on a machine with CUDA 10.x.
• Save intermediate artifacts in a S3 bucket: If tests pass, then we can deploy built artifacts immediately.

Add Windows target

The progress has been slow on this front. The main challenge is to get Jenkins to somehow launch Windows workers and send remote commands. We've run into issues compiling XGBoost on Windows a few times (#4139, #3869), so it would be nice to detect potential issues early on. In addition, we want to build Python wheels automatically (I've been building the Windows wheel manually).

Migrate Python and Java tests to Jenkins

• The Python tests in the Travis service tend to fail quite often, see Roadmap: Fix flaky tests #3720. Move these tests to Jenkins, where we can choose workers with larger memory.
• As discussed in [jvm-packages] Spylon-kernel java.lang.ClassCastException: ml.dmlc.xgboost4j.scala.Booster cannot be cast to ml.dmlc.xgboost4j.scala.Booster #4218, we want to properly support both Spark 2.3.x and 2.4.x. To that end, try building XGBoost4J-Spark with Spark 2.3.x and run it on machine with Spark 2.4.x (and vice versa).

Regular performance tests

We should run a suite of performance tests on a regular basis (say every two weeks). This way, we can detect any performance degradation.

@dmlc/xgboost-committer

[hcho3]

ETA for the first pull request for this RFC: end of this week (March 15, 2019)

[hcho3]

It turns out that it is possible to compile CUDA code on machines without NVIDIA GPUs:

• https://stackoverflow.com/questions/20186848/can-i-compile-a-cuda-program-without-having-a-cuda-device
• MXNet uses CPU workers to compile CUDA code:
  https://github.com/apache/incubator-mxnet/blob/0e8c27096289fcb77e473471df99e16e5aa242b9/ci/jenkins/Jenkins_steps.groovy#L164-L176

The benefit is that we can use more powerful CPU instances (e.g. c5d.18xlarge) to compile CUDA code faster.

cc @trivialfis @RAMitchell

[terrytangyuan]

Definitely +1 for running performance tests regularly. Probably more frequent than two weeks though.

[hcho3]

@terrytangyuan Any suggestions for performance tests?

[terrytangyuan]

@terrytangyuan Any suggestions for performance tests?

Not on top of my head. I've only done this for internal datasets before. But here we should probably pick some good public (Kaggle?) datasets. Note that performance tests include both statistical and computational performance so we may want to consider use cases from both perspectives.

[hcho3]

@terrytangyuan Thanks. Let me think over this over the weekend.

[RAMitchell]

Would something like this be okay for performance benchmarking? We have a more polished Nvidia version that could be open sourced. All of the dataset loading/processing is automatic and it runs on a docker container.

[trivialfis]

Preferably move R tests to Jenkins where we can cache all built dependencies. ;-)

[hcho3]

@trivialfis Yes, yes, yes! Docker is a great invention