Retry on CondaSSLError.

[bdice]

Description

This PR will retry conda commands on CondaSSLError.

Context

We have recently seen more SSL errors, like those we attempted to fix in rapidsai/cuml#6177 and rapidsai/cugraph#4825. However, we still observe SSL errors while downloading conda packages.

Errors look like this:

CondaSSLError: Encountered an SSL error. Most likely a certificate verification issue.

Exception: [SYS] unknown error (_ssl.c:2580)

In conda/conda#11564, CondaSSLError was split from CondaHTTPError. This was released in conda 4.14.0.

Currently it appears that all the errors in cuML's CI (sample nightly run) are either CondaSSLError or actual test failures (some are from Hypothesis, some are not). I hope this PR eliminates the CondaSSLError cases.

Alternatives

From https://stackoverflow.com/questions/31729076/conda-ssl-error, it seems there may be alternative solutions. I think we should add this retry either way, but we may be able to do more.

• Setting ssl_verify: false in the conda config
• Maybe we need to add/set a root certificate in the CI images?
• Maybe we need to install a newer certifi in the ci-conda images? The preinstalled version is currently 2024.8.30 and running mamba update -n base certifi updates it to 2024.12.14. There could be certificate changes in those 4 months.
  • Testing this here: Add certifi to CI images ci-imgs#218
• More ideas: https://docs.conda.io/projects/conda/en/stable/user-guide/configuration/non-standard-certs.html

[jameslamb]

Thanks for working on this and for the thorough description!

I agree, we should add this retry logic. I've also observed that these are often resolved by a retry, which suggests that "certificates in the CI images are out of date" is not the only cause of this.

[jakirkham]

On the Conda side, certificates come from the ca-certificates package. This does repackage the certificates from certifi, but bundles them separate from any Python code

Also Conda does have a (configurable) list of packages it always checks for updates called, aggressive_update_packages, which includes ca-certificates

Note there was an update to to ca-certificates this weekend: conda-forge/ca-certificates-feedstock#70