Implementing conda lock

[valeriupredoi]

Description

A fast conda environment creation is possible using the provided conda lock files.
To build a conda lock file install the conda-lock package first, then
run conda-lock lock --platform linux-64 -f environment.yml --mamba
(mamba activated for speed) to create a conda lock file for Linux platforms,
or run conda-lock lock --platform osx-64 -f environment.yml --mamba to create a lock file
for OSX platforms. A conda lock file is an explicit environment file that contains pointers to
dependency packages as they are hosted on the Anaconda cloud; these have frozen version numbers
and build hashes, established at the time of the conda lock file creation, so may be obsolete after a while,
but they allow for a robust environment creation while they're still up-to-date. We regenerate these lock files every 10 days, so to minimize the risk of dependencies becoming obsolete. After the lock file has been created,
conda environment creation then happens as per normal use with e.g.
conda create --name esmvaltool --file conda-linux-64.lock (NOTE that pip and conda are NOT installed
in this environment so manual installation is needed, conda install -c conda-forge pip at the very
minimum so we can install esmvalcore).

Functionality

Currently the lock file works fine on Linux, we have a dedicated build and verify Github Actions workflow that runs every 10 days. The lock files are created and verified every 10 days at 4am, the choice of hour is so that it doesn't conflict with the GA test that runs nightly and performs the installation and test from the conda lock file (that runs at midnight). Upon (re-)creation of the lock files an automated PR is generated e.g. this one if the lock file has changed in any way. Note that the auto PR will have me assigned and reviewer, and will only include one changed file, the lock file.

Benefits

Once we have a conda lock file, the conda env creation is blindingly fast (order 2min), and this can be used as a lifeboat if we have issues with the current solving, the env from the lock file being at most 10 days old, so really not too bad. Also the lockfile it serves as a reference for fallbacks, if an environment breaks down due to a dependency being updated and entering conflict with one or more other dependencies.

What's working

Linux OS with latest Python (3.10) conda lock file generation, env creation from it, package installation and testing, auto PR generation when the lock file has changed.

What's not working

OSX: we still have to understand what's going on in OSX land - all sorts of issues related to both lock file generation and env creation from it.

Closes #1139

[valeriupredoi]

hmmm it seems that the conda incubator can't generate an env from an explicit lock file, I asked the guys there in conda-incubator/setup-miniconda#178 - chances are I'm doing something silly

[valeriupredoi]

OK progress! we have a working conda lock file construction and full installation and testing from it for linux-64 🥳 - see the GA tests below, I need to figure out what's that issue with that stubs lib - @stefsmeets you might wanna have a look at this issue since it's the mypy's that are fainting. I'll examine locally too (although at first pass I don't get that issue locally)

[stefsmeets]

OK progress! we have a working conda lock file construction and full installation and testing from it for linux-64 🥳 - see the GA tests below, I need to figure out what's that issue with that stubs lib - @stefsmeets you might wanna have a look at this issue since it's the mypy's that are fainting. I'll examine locally too (although at first pass I don't get that issue locally)

I see that the installation part of the test is now passing after #1167 got merged. 😁

[valeriupredoi]

Indeed! @bouweandela and myself were figuring out the fails in parallel, himself looking at Circle and me gawking at the GA tests. Redundancy FTW man (we're not like Fastly) 😁

[bvreede]

Locally running the commands

conda install -y conda-lock -c conda-forge

to install conda-lock; and then

conda-lock lock --platform osx-64 -f environment.yml --mamba

to generate the conda lockfile, does not create issues for me. Can you give more info about what's up with OSX here, @valeriupredoi?

[zklaus]

I think we started looking at the conda lock files due to very long resolution times for conda. I think with mamba that is mitigated to some degree, but I am thinking more and more that those lock files could play a role in reproducibility.

[valeriupredoi]

OK gents, cheers for yer input! I have now added a mention to why conda lock - an alternative to source install if the conda env can not be created for some reason, what you thinks?

[valeriupredoi]

@bouweandela a link to what? The conda lock file? Can't add that until it's merged, plus it will be overwhelming if someone opens that file 😁

[valeriupredoi]

this is now fully (and beautifully) working, including an automated PR see #1424 🥳

[valeriupredoi]

@zklaus got a bit of time to have a looksee&review of this, please, mate? I'd be rather happy if this got in 2.5 😁

[zklaus]

Very minor comments. Nice job.

[valeriupredoi]

very many thanks @zklaus 🍺 I have checked and all @bouweandela 's review suggestions have been addressed so I'm gonna dismiss his review (although that sounds harsh, sorry Bouwe - don't mean to, but all's working fine and it's good to have this in!) 🥳

all points raised by Bouwe have been addressed, cheers Bouwe

[valeriupredoi]

d'you know what I realized? This little bugger will email me at 4am everytime it opens a PR with the changed conda lock file, I might have to change the hour for the cron job 🤣

[bouweandela]

@bouweandela a link to what? The conda lock file? Can't add that until it's merged, plus it will be overwhelming if someone opens that file grin

My apologies for being slow to reply. Indeed I meant a link to where this file can be downloaded, because this is not clear now in the documentation.

[valeriupredoi]

sounds like a good plan, Bouwe, lemme just crack open a PR with that!

[valeriupredoi]

Bouwe dunnit in #1462