Fix quantile tests running on multi-gpus

[rongou]

As reported in #8710 (comment).

Closes #8782 .

[rongou]

@trivialfis @hcho3 I'm surprised these tests are not run in the mgpu buildkite pipeline. Do we need to enable them?

[hcho3]

We run all gtests using a single GPU only. This was done to save CI cost.

[rongou]

Ah I see. So what does the multi-gpu pipeline do? :)

[hcho3]

@rongou It runs Python tests with mgpu markers

[rongou]

Can we also run the multi-gpu c++ tests there? For example, we can name the two tests here something like MGPUQuantile, and then it's a matter of running testxgboost --gtest_filter=MGPU*.

[hcho3]

@rongou That's one option. Alternatively, we can create a separate gtest binary for the multi-GPU tests.

[trivialfis]

The CPP MGPU tests were removed after having dask tests. However, it seems we might need to bring it back as some of them are moved back to CPP from dask with the development of federated learning.

[trivialfis]

Opened an issue for tracking #8782 .

[hcho3]

See my latest commit. It will run the quantile tests using 4 GPUs

[trivialfis]

Thank you for the quick CI update @hcho3 .

@rongou Could you please fix the error on the new CI test?

[rongou]

@hcho3 is this a transient error or something wrong with the configuration?

An error occurred (RepositoryAlreadyExistsException) when calling the CreateRepository operation: The repository with name 'xgb-ci.gpu11.0' already exists in the registry with id '492475357299'

[hcho3]

@rongou The error is expected. It means that the Docker cache already has the Docker image for this CI pipeline. Can you take a look and find out why the gtest binary is crashing?

[rongou]

@hcho3 is there any way to get more information from the run? Right now it just says

Error: The command exited with status 135

I can't reproduce this on my local machine with 2 GPUs.

[hcho3]

Is it using NCCL?

[rongou]

Yes.

[hcho3]

I'll try to debug on my end