Parallel builds of environments

[costrouc]

A conda environment install can be broken into several steps:

1. solve environment from environment.yaml
2. download and extract packages to package cache
3. install conda environment

Currently all of this is done in https://github.com/Quansight/conda-store/blob/main/conda-store-server/conda_store_server/build.py#L97-L110 via conda env create -f environment.yaml -p <prefix> in one monolithic approach. As a first approach (which can be improved later) we need to break this function into several steps.

Replace conda env create -f environment.yaml -p <prefix> with

1. conda-lock -f environment.yml -p linux-64 this will generate a yaml file for the lock
2. Python code to inspect the conda lock yaml file (example here) and download each package with manager: conda to the conda package cache directory and extract. This step must use locking using filelock https://github.com/Quansight/conda-store/blob/main/conda-store-server/conda_store_server/build.py#L135-L137.
3. conda-lock install -p <prefix>

[costrouc]

You can get the package cache directory via the following. This will help determine where packages are downloaded an extracted.

from conda.base.context import context
context.pkgs_dirs

There are two formats that I know of for packages that are downloaded and need to be handled.

• .tar.gz (straightforward to handle)
• .conda https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/packages.html#conda-file-format

[costrouc]

@aktech was there any progress in implementing this? Happy to meet and discuss further if this is still a priority.

[costrouc]

Closing as implemented in #417

[costrouc]

See notes form a matrix conversation https://matrix.to/#/!SgckMCYyQUhdyooKZe:gitter.im/$U5CYJdRgHFBF6pqWSmP_h2HbY8mHTTl7wX-F1Po1JkI?via=gitter.im&via=matrix.org&via=tum.de