Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document installation procedure for environments without Internet #302

Open
sgouache opened this issue Dec 5, 2024 · 1 comment
Open
Labels
Status/Draft The issue is still not well defined

Comments

@sgouache
Copy link

sgouache commented Dec 5, 2024

Description

In some secure or broken environments, making requests to the Internet is not possible/permitted.
This issue proposes to define an installation procedure for such environments.
More specifically it tries to address the case of GCP notebooks - jupyter kernel running a conda environment - but described procedure could be applied to other situations.

Proposed idea

Build phase: build installation package for target environment

0. create build environment identical to target (this can be done manually or in a GH pipeline)

docker run -v $PWD:/mnt -it gcr.io/deeplearning-platform-release/base-cpu:latest bash

Inside docker container, run the following to
1. install khiops

conda install -c conda-forge -c khiops khiops -y

2. create a snapshot of installed packages

cd /mnt
conda list -n base --explicit > packages.wget

3. download all packages to allow offline installation

mkdir -p pkgs && cd pkgs
wget -i ../packages.wget
cd ..

4. update package list to point at local version (this replaces all http URIs with local paths)

awk -F '/' '/^http/ {print "pkgs/"$NF; next} {print} ' packages.wget > khiops-local.yaml

5. create a zip for easier transportation to the target - outside container

zip -r khiops-local.zip khiops-local.yaml pkgs

Deploy phase

1. Upload the zip file to the target machine, notebook etc.
2. unzip the contents

unzip khiops-local.zip

3a. install contents into existing conda environment - this might replace some packages that are already present

conda install --file khiops-local.yaml --offline

3b. alternative: create new environment with the contents of the archive

conda create -n my_env --file khiops-local.yaml --offline

The above has been tested with khiops 10.2.3 and deployed sucessfully on GCP notebooks with no Internet access.
One thing to note is the fairly big size of the zip file: 149M. It's big, but because it contains everything needed to run the conda environment, it's almost guaranteed to work.
Feedbacks, simplifications, alternative proposals welcome.

@popescu-v
Copy link
Collaborator

Consider using conda pack for the first stage of the workflow?

@folmos-at-orange folmos-at-orange changed the title Document installation procedure for environments where Internet connectivity is unavailable Document installation procedure for environments without Internet Dec 11, 2024
@folmos-at-orange folmos-at-orange added the Status/Draft The issue is still not well defined label Dec 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Status/Draft The issue is still not well defined
Projects
None yet
Development

No branches or pull requests

3 participants