Use gpuci_mamba_retry on Java CI.

[bdice]

Resolves #9976.

[codecov]

Codecov Report

Merging #9983 (a879033) into branch-22.02 (967a333) will decrease coverage by 0.03%.
The diff coverage is n/a.

❗ Current head a879033 differs from pull request most recent head 630b584. Consider uploading reports for the commit 630b584 to get more accurate results

@@               Coverage Diff                @@
##           branch-22.02    #9983      +/-   ##
================================================
- Coverage         10.49%   10.45%   -0.04%     
================================================
  Files               119      119              
  Lines             20305    20417     +112     
================================================
+ Hits               2130     2134       +4     
- Misses            18175    18283     +108     

Impacted Files	Coverage Δ
python/custreamz/custreamz/kafka.py	29.16% <0.00%> (-0.63%)	⬇️
python/dask_cudf/dask_cudf/sorting.py	92.30% <0.00%> (-0.61%)	⬇️
python/cudf/cudf/__init__.py	0.00% <0.00%> (ø)
python/cudf/cudf/core/frame.py	0.00% <0.00%> (ø)
python/cudf/cudf/core/index.py	0.00% <0.00%> (ø)
python/cudf/cudf/io/parquet.py	0.00% <0.00%> (ø)
python/cudf/cudf/core/series.py	0.00% <0.00%> (ø)
python/cudf/cudf/utils/utils.py	0.00% <0.00%> (ø)
python/cudf/cudf/utils/dtypes.py	0.00% <0.00%> (ø)
python/cudf/cudf/utils/ioutils.py	0.00% <0.00%> (ø)
... and 17 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 23603d1...630b584. Read the comment docs.

[ajschmidt8]

Hmmm. The good news is that mamba doesn't seem to be downgrading the packages anymore as I mentioned here. Looks like we're still seeing that same Could NOT find JNI (missing: JAVA_JVM_LIBRARY) error though.

[bdice]

CI is failing because CMake cannot find the JNI despite openjdk being installed.

The issue is arising because conda install "openjdk=8.*" runs the activation script included in the package upon installation, while mamba install "openjdk=8.*" is not able to run that activation script on installation. That means that conda install is able to set the environment variable JAVA_HOME, while mamba does not. To get this to work, we'll need to reactivate the environment or manually set JAVA_HOME=$CONDA_PREFIX.

Bash script to reproduce the issue:

#!/bin/bash

# Enable activating conda environments
. $CONDA_PREFIX/etc/profile.d/conda.sh

export MAMBA_NO_BANNER=1

echo "---------conda---------"
conda --version
conda create -y -q -n conda_jdk_test
conda activate conda_jdk_test
conda install -y -q "openjdk=8.*"
echo "'which java' returns $(which java)"
echo "JAVA_HOME=${JAVA_HOME}"

conda deactivate
conda env remove --name conda_jdk_test
echo "-------end conda-------"
echo

echo "---------mamba---------"
mamba --version
mamba create -y -q -n mamba_jdk_test mamba
conda activate mamba_jdk_test
mamba install -y -q "openjdk=8.*"
echo "'which java' returns $(which java)"
echo "JAVA_HOME=${JAVA_HOME}"  # This is the bug -- JAVA_HOME is not set.
conda deactivate

echo "Retry mamba after re-activating:"

conda activate mamba_jdk_test
mamba install -y -q "openjdk=8.*"
echo "'which java' returns $(which java)"
echo "JAVA_HOME=${JAVA_HOME}"  # This is the bug -- JAVA_HOME is not set.
conda deactivate

conda env remove --name mamba_jdk_test
echo "-------end mamba-------"

Simplified output:

---------conda---------
conda 4.11.0

[... creating / installing ...]
Executing transaction: ...working... done
'which java' returns /home/bdice/mambaforge/envs/conda_jdk_test/bin/java
JAVA_HOME=/home/bdice/mambaforge/envs/conda_jdk_test
Remove all packages in environment /home/bdice/mambaforge/envs/conda_jdk_test:
-------end conda-------

---------mamba---------
mamba 0.19.1
conda 4.11.0

[... creating / installing ...]
Executing transaction: ...working... done
'which java' returns /home/bdice/mambaforge/envs/mamba_jdk_test/bin/java
JAVA_HOME=
Retry mamba after re-activating:
'which java' returns /home/bdice/mambaforge/envs/mamba_jdk_test/bin/java
JAVA_HOME=/home/bdice/mambaforge/envs/mamba_jdk_test
Remove all packages in environment /home/bdice/mambaforge/envs/mamba_jdk_test:
-------end mamba-------

It appears that this PR is related: conda/conda#10783 and particularly this comment: conda/conda#10783 (comment)

The problem in mamba is the following:

• the environment has to be re-activated after an "install" command to properly execute activation scripts, hash the binaries on the PATH etc.

[bdice]

Thanks @jlowe @ajschmidt8 @raydouglass for the pointers -- Java CI is now passing, this is ready for review and merge. ☕ 🚀

cc: @shwina @sameerz

[jlowe]

Thanks @bdice for driving this!

[sameerz]

Thank you @bdice !

[ajschmidt8]

Good find! I wonder if there's an upstream mamba bug about this that we can track so that we can eventually remove this workaround.

[bdice]

Good find! I wonder if there's an upstream mamba bug about this that we can track so that we can eventually remove this workaround.

@ajschmidt8 There’s a conda issue that I Iinked above: conda/conda#10783 The issue is that mamba cannot currently hook into conda to trigger the post-install activation.

[bdice]

@gpucibot merge

[bdice]

For the sake of completeness, here's an analysis of the CI runtime before/after this PR. Comparing the CI build logs, I see a significant improvement in run time by switching to mamba.

CI Log	Before (conda, commit a4dc42d), CI log	After (mamba, commit 630b584), CI log
Total Runtime	0:58:59	0:11:42
Install dependencies	0:22:06	0:00:50
Installing libcudf and libcudf_kafka	0:27:21	0:01:04