How to set up conda env cache in GitLab CI/CD using micromamba?

[bgalvao]

• GitLab CI/CD seems to offer a solution specifically for pip's cache and currently offers no solution for conda.
• Benjamin Bertrand found a solution by using a docker volume with a self-hosted gitlab-runner

Is there any way to do this with GitLab's public gitlab-runners?

[mpizenberg]

I’m very much interested in guidelines for this too. Currently I’m thinking of using gitlab container registry with micromamba docker. On issue I have is that I’d like to be able to have mamba root prefix within the gitlab build directory because caching does not work outside of it. I have a setup where I use a combination of micromamba and pip-tools. So I’m looking for a way to have a dockerfile in which the mamba environment is already created, and a way to have it in gitlab build directory when executing gitlab-ci yaml file. I’m trying to simulate this situation by a mv /old_prefix /new_prefix folowed by export $MAMBA_ROOT_PREFIX=/new_prefix, activating environment, and finally calling pip-sync to finalize with the python packages dependencies. All seems to be working all right but something is off. For example, I have pyright in my requirements, and yet after the pip-sync it’s not available in the command line.

Any guideline on how to successfully change location of a micromamba environment, specifically for CI?

[maresb]

Conda environments use wrapper scripts with hard-coded paths, so a Conda environment will be broken if you move it to a different directory.

Can you simply use the Docker cache? The initial setup somehow feels more complicated than it should be, but basically you just create a new image for caching in the GitLab container registry, and then in your docker build command just add --cache-to type=registry,ref=<registry>/<cache-image> --cache-from type=registry,ref=<registry>/<cache-image>. That way things will only be rebuilt if you change your environment file (or if your cache busts before that step).

It's possible you'll need to tweak a few further things which I'm not aware of, since my experience in this realm has been with either docker-compose or buildah, and not docker build itself.

[mpizenberg]

Oh I was not aware of this docker feature. Thanks for the link and explanation. I suppose I have a lot more reading to do.