diff --git a/doc/guide/Guide_Euler.ipynb b/doc/guide/Guide_Euler.ipynb index 2f2cfa7f4..ccfc8a445 100644 --- a/doc/guide/Guide_Euler.ipynb +++ b/doc/guide/Guide_Euler.ipynb @@ -26,7 +26,7 @@ }, "source": [ "\n", - "## Installation and working directories\n", + "## Installation directory and working directory\n", "\n", "Please, get familiar with the various Euler storage options: https://scicomp.ethz.ch/wiki/Storage_systems.
\n", "As a general rule: use `/cluster/project` for installation and `/cluster/work` for data processing.\n", @@ -44,10 +44,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\n", - "## Pre-installed version of Climada\n", - "\n", - "Climada is pre-installed and available in the default pip environment of Euler." + "## Climada installation in a virtual environment" ] }, { @@ -57,12 +54,24 @@ "\n", "### 1. Load dependencies\n", "\n", - "Use the new software stack. Unless you have already done so, run `set_software_stack.sh new`.\n", - "\n", "```bash\n", - "module load gcc/8.2.0 python/3.11.2 hdf5/1.10.1 gdal/3.4.3 geos/3.9.1 proj/8.2.1 libspatialindex/1.9.3 netcdf/4.6.3 eccodes/2.31.0 zlib/1.2.9 libtiff/4.2.0 sqlite/3.35.5\n", + "module load \\\n", + " gcc/12.2.0 \\\n", + " stack/2024-06 \\\n", + " python/3.11.6 \\\n", + " hdf5/1.14.3 \\\n", + " geos/3.9.1 \\\n", + " sqlite/3.43.2 \\\n", + " eccodes/2.25.0 \\\n", + " gdal/3.6.3 \\\n", + " eth_proxy\n", + "\n", + "module load proj\n", + "module unload proj\n", "```\n", "\n", + "(The last two lines may seem odd but they are working around a conficting dependency version situation.)\n", + "\n", "You need to execute this every time you login to Euler before Climada can be used. \n", "To safe yourself from doing it manually, append these lines to the ~/.bashrc script, which is automatically executed upon logging in to Euler." ] @@ -72,25 +81,73 @@ "metadata": {}, "source": [ "\n", - "### 2. Check installation \n", + "### 2. Create and prepare virtual environment\n", "\n", "```bash\n", - "python -c 'import climada; print(climada.__file__)'\n", - "```\n", + "envname=climada_env\n", + "\n", + "# create environment\n", + "python -m venv --system-site-packages /cluster/project/climate/$USER/venv/$envname\n", + "\n", + "# acitvate it\n", + ". /cluster/project/climate/$USER/venv/$envname/bin/activate\n", "\n", - "should output something like this:\n", + "# install python kernel (to be used in JupyterHub, s.b.)\n", + "python -m ipykernel install --user --name $envname\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 3. Install dependencies\n", "\n", "```bash\n", - "/cluster/apps/nss/gcc-8.2.0/python/3.10.4/x86_64/lib64/python3.10/site-packages/climada/__init__.py\n", + "pip install \\\n", + " dask[dataframe] \\\n", + " fiona==1.9 \\\n", + " gdal==3.6 \\\n", + " netcdf4==1.6.2 \\\n", + " rasterio==1.4 \\\n", + " pyproj==3.7 \\\n", + " geopandas==1.0 \\\n", + " xarray==2024.9 \\\n", + " sparse==0.15\n", "```" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 4. Install climada\n", + "\n", + "There are two options. Either install from the downloaded repository (option A), or use a particular released version (option B).\n", + "\n", + "#### option A\n", + "```bash\n", + "cd /cluster/project/climate/$USER # or wherever you plan to download the repository\n", + "git clone https://github.com/CLIMADA-project/climada_python.git # unless this has been done before\n", + "cd climada_python\n", + "pip install -e .\n", + "```\n", + "\n", + "#### option B\n", + "\n", + "```bash\n", + "pip install climada=5.0\n", + "```\n", + "\n", + "or whatever version you prefer" + ] + }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", - "### 3. Adjust the Climada configuration\n", + "### 5. Adjust the Climada configuration\n", "\n", "Edit a configuration file according to your needs (see [Guide_Configuration](../guide/Guide_Configuration.ipynb)).\n", "Create a climada.conf file e.g., in /cluster/home/$USER/.config with the following content:\n", @@ -112,35 +169,25 @@ "metadata": {}, "source": [ "\n", - "### 4. Run a job\n", + "### 6. Test the installation\n", "\n", - "Please see the docs at https://slurm.schedmd.com/ on how to use the `slurm` batch system \n", - "and the Wiki https://scicomp.ethz.ch/wiki/Transition_from_LSF_to_Slurm for a mapping of `lsf` commands to their `slurm` equivalents.\n", + "Check installation in login node:\n", "\n", "```bash\n", - "cd /cluster/work/climate/$USER # change to the working directory\n", - "sbatch [slurm options*] --wrap 'python climada_job_script.py' # submit the job\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ + "python -m unittest climada.engine.test.test_impact_calc\n", + "```\n", "\n", - "## Working with Git branches\n", + "This should prompt the usual \"`OK`\" in the end.\n", + "Once that succeeded you may want to test the installation also in a compute node, just for the sake of it:\n", "\n", - "If the Climada version of the default installation is not according to your needs, you can install Climada from a local Git repository." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ + "```bash\n", + "sbatch --wrap=\"python -m unittest climada.engine.test.test_impact_calc\"\n", + "```\n", "\n", - "### 1. Load dependencies \n", + "Look for the \"`OK`\" in the hereby created `slurm-[XXXXXXX].out` file\n", "\n", - "See [Load dependencies](#1.-load-dependencies) above." + "Please see the docs at https://slurm.schedmd.com/ on how to use the `slurm` batch system \n", + "and the Wiki https://scicomp.ethz.ch/wiki/Transition_from_LSF_to_Slurm for a mapping of `lsf` commands to their `slurm` equivalents." ] }, { @@ -148,67 +195,52 @@ "metadata": {}, "source": [ "\n", - "### 2. Create installation environment\n", - "\n", - "```bash\n", - "python -m venv --system-site-packages /cluster/project/climate/$USER/climada_venv\n", - "```" + "## Run a Jupyter Notebook on Euler" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ + "It is possible to run a Jupyter Notebook on Euler within a JupytherHub instance running as an interactive slurm job.\n", + "See the documentation https://scicomp.ethz.ch/wiki/JupyterHub.\n", "\n", - "### 3. Checkout sources\n", + "For using climada inside the jupyter notebook, you need to create a customized `jupyterlabrc` file by running the following lines:\n", "\n", "```bash\n", - "cd /cluster/project/climate/$USER\n", - "git clone https://github.com/CLIMADA-project/climada_python.git\n", - "cd climada_python\n", - "git checkout develop # i.e., your branch of interest\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ + "mkdir -p ~/.config/euler/jupyterhub\n", + "cat > ~/.config/euler/jupyterhub/jupyterlabrc < **WARNING:** This approach is highly discouraged, as it imposes a heavy and mostly unnecessary burden on the file system of the cluster." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Installation\n", + "- Make sure your python environment is activated.\n", + "- Run `pip install --upgrade MISSING_MODULE`.\n", "\n", - "#### 1. Conda\n", + "### 2. Upgrading from Python 3.9 or 3.10\n", "\n", - "Download or update to the latest version of [Miniconda](https://conda.io/miniconda.html).
\n", - "Installation is done by execution of the following steps:\n", + "Virtual environments created are i.g. only working for the Python version they were created with.\n", + "In particular Python kernels from 3.9 environments will fail to connect in a Jupyter notebook on https://jupyter.euler.hpc.ethz.ch/.\n", "\n", - "```bash\n", - "cd /cluster/project/climate/USERNAME\n", - "wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh\n", - "bash Miniconda3-latest-Linux-x86_64.sh\n", - "miniconda3/bin/conda init\n", - "rm Miniconda3-latest-Linux-x86_64.sh\n", - "```\n", + "- It's suggested to create new environments and remove the old kernels from `~/.local/share/jupyter/kernels/`.\n", "\n", - "During the installation process of Miniconda, you are prompted to set the working directory according to your choice.\n", - "Set it to `/cluster/project/climate/USERNAME/miniconda3`.
\n", - "Once the installation has finished, log out of Euler and in again. The command prompt should be preceded by `(base)`, \n", - "indicating that the installation was a success and that you login in into conda's base environment by default." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### 2. Checkout sources \n", + "### 3. Incompatible GEOS version\n", "\n", - "See [Checkout sources](#3.-Checkout-sources) above." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### 3. Climada environment\n", + "If you get a warning `UserWarning: The Shapely GEOS version (3.9.1-CAPI-1.14.2) is incompatible with the GEOS version PyGEOS was compiled with (3.9.1-CAPI-1.14.2). Conversions between both will be slow.` or similar (version numbers may vary), updating geopandas can help:\n", "\n", - "Create the conda environment:\n", + "- Create and activate a virtual environment with `venv` (s.a.)\n", + "- Run `pip install --upgrade geopandas`\n", "\n", - "```bash\n", - "cd /cluster/project/climate/USERNAME/climada_python\n", - "conda env create -f requirements/env_climada.yml --name climada_env\n", - "conda env update -n climada_env -f requirements/env_developer.yml\n", + "### 4. Installation doesn't work\n", "\n", - "conda activate climada_env\n", - "conda install conda-build\n", - "conda develop .\n", - "```" + "If you have additional requirements it may be that the installation process described above is failing. In this case you can run climada from a customized singularity container." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "#### 4. Adjust the Climada configuration\n", + "## Fall back: Singularity Container\n", + "\n", + "In case the installation in a virtual environment does not work, e.g,, because some module on Euler in incompatible with additional requirements for Python packages, the last resort is an installation of CLIMADA into a Singularity container. \\\n", + "In general, this is more difficult and time-consuming and easier to get wrong. It also requires a lot of diskspace and produces a high number of files, but it provides more flexibility, as one can install basically anything you want.\n", "\n", - "See [Adjust the Climada configuration](#3.-Adjust-the-Climada-configuration) above." + "To install CLIMADA into a Singularity container, follow these steps:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "#### 5. Climada scripts \n", + "### 1. create a recipe\n", + "\n", + "Create a file `recipe.txt` with the following content:\n", "\n", - "Create a bash script for executing python scripts in the climada environment, `climadajob.sh`:\n", - "```bash\n", - "#!/bin/bash\n", - "PYTHON_SCRIPT=$1\n", - "shift\n", - ". ~/.bashrc\n", - "conda activate climada_env\n", - "python $PYTHON_SCRIPT $@\n", - "echo $PYTHON_SCRIPT completed\n", "```\n", + "Bootstrap: docker\n", + "From: nvidia/cuda:12.0.0-devel-ubuntu22.04\n", "\n", - "Make it executable:\n", "\n", - "```bash\n", - "chmod +x climadajob.sh\n", - "```\n", + "%labels\n", + " version=\"1.0.0\"\n", + " description=\"climada\"\n", "\n", - "Create a python script that executes climada code, e.g., `climada_smoke_test.py`:\n", "\n", - "```python\n", - "import sys\n", - "from climada import CONFIG, SYSTEM_DIR\n", - "from climada.util.test.test_finance import TestNetpresValue\n", - "TestNetpresValue().test_net_pres_val_pass()\n", - "print(SYSTEM_DIR)\n", - "print(CONFIG.local_data.save_dir.str())\n", - "print(\"the script ran with arguments\", sys.argv)\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### 6. Run a job \n", + "%post\n", "\n", - "With the scripts from above you can submit the python script as a job like this:\n", + " # Install requirements\n", + " apt-get -y update\n", + " DEBIAN_FRONTEND=\"noninteractive\" TZ=\"Europe/Rome\" apt-get -y install tzdata\n", + " apt-get install -y `apt-cache depends openssh-client | awk '/Depends:/{print$2}'`\n", + " apt-get download openssh-client\n", + " dpkg --unpack openssh-client*.deb\n", + " rm /var/lib/dpkg/info/openssh-client.postinst -f\n", + " dpkg --configure openssh-client\n", + " apt-get -y install tk tcl rsync wget curl git patch\n", "\n", - "```bash\n", - "sbatch [slurm options] --wrap \"/path/to/climadajob.sh /path/to/climada_smoke_test.py arg1 arg2\"\n", - "```\n", + " mkdir -p /opt/software\n", "\n", - "After the job has finished the slurm output file should look something like this:\n", + " # Install conda and mamba\n", + " cd /opt/software\n", + " curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh\n", + " sh ./Miniconda3-latest-Linux-x86_64.sh -p /opt/software/conda -b\n", + " /opt/software/conda/bin/conda install -y -c conda-forge mamba\n", "\n", - "```\n", - "/cluster/work/climate/USERNAME/climada/data\n", - "/cluster/work/climate/USERNAME/climada/results\n", - "the script ran with arguments ['/path/to/climada_smoke_test.py', 'arg1' 'arg2']\n", - "python_script.sh completed\n", - "```\n", + " # Create and activate environment\n", + " /opt/software/conda/bin/mamba create -n climada_env python=3.11 --yes\n", + " . /opt/software/conda/etc/profile.d/conda.sh && conda activate climada_env\n", "\n", - "Please see the docs at https://slurm.schedmd.com/ on how to use the `slurm` batch system \n", - "and the Wiki https://scicomp.ethz.ch/wiki/Transition_from_LSF_to_Slurm for a mapping of `lsf` commands to their `slurm` equivalents." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Deinstallation" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### 1. Conda \n", + " # Install jupyter\n", + " python -m pip install jupyterhub jupyterlab\n", "\n", - "Remove the miniconda3 directory from the installation directory:\n", "\n", - "```bash\n", - "rm -rf /cluster/project/climate/USERNAME/miniconda3/\n", - "```\n", + " # Install climada from source\n", + " mkdir -p /opt/climada_workspace\n", + " cd /opt/climada_workspace\n", "\n", - "Delete the conda related parts from `/cluster/home/USERNAME/.bashrc`, i.e., everything between\n", + " git clone https://github.com/CLIMADA-project/climada_python.git\n", + " cd climada_python\n", + " git checkout develop\n", "\n", - "`# >>> conda initialize >>>`\\\n", - "and\\\n", - "`# <<< conda initialize <<<`" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### 2. Climada\n", + " mamba env update -n climada_env -f requirements/env_climada.yml\n", + " python -m pip install -e ./\n", "\n", - "Remove the climada sources and config file:\n", + " # Install climada-petals from source\n", + " cd /opt/climada_workspace\n", "\n", - "```bash\n", - "rm -rf /cluster/project/climate/USERNAME/climada_python\n", - "rm -f /cluster/home/USERNAME/climada.conf /cluster/home/USERNAME/*/climada.conf\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ + " git clone https://github.com/CLIMADA-project/climada_petals.git\n", + " cd climada_petals\n", + " git checkout develop\n", "\n", - "## Run Jupyter Notebook on Euler" + " mamba env update -n climada_env -f requirements/env_climada.yml\n", + " python -m pip install -e ./\n", + "\n", + "\n", + "%environment\n", + " #export LC_ALL=C\n", + "\n", + "\n", + "%runscript\n", + " . /opt/software/conda/bin/activate && conda activate climada_env\n", + " $@\n", + "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "It is possible to run a Jupyter Notebook on Euler within a JupytherHub instance running as an interactive slurm job.\n", - "See the documentation https://scicomp.ethz.ch/wiki/JupyterHub.\n", + "### 2. build the container\n", "\n", - "For using climada inside the jupyter notebook, create a `.jupyterlabrc` file in your Euler home directory with the following content:\n", + "- as it is cpu and memory consuming, run a job\n", + "- the container will be created in the directory `climada.sif`, it is going to be huge, so it's best located within the `project` file system\n", "\n", - "```\n", - "module purge\n", - "module load StdEnv gcc/8.2.0 python_gpu/3.11.2 eth_proxy hdf5/1.10.1 gdal/3.4.3 geos/3.9.1 proj/8.2.1 libspatialindex/1.9.3 netcdf/4.6.3 eccodes/2.31.0 zlib/1.2.9 libtiff/4.2.0 sqlite/3.35.5\n", - "```\n", - "\n", - "Then login to https://jupyter.euler.hpc.ethz.ch/ and start a JupyterLab server." + "```bash\n", + "sbatch \\\n", + " --ntasks=1\\\n", + " --cpus-per-task=1 \\\n", + " --time=1:00:00 \\\n", + " --job-name=\"build-climada-container\" \\\n", + " --mem-per-cpu=4096 \\\n", + " --wrap=\"singularity build --sandbox /cluster/project/[path/to]/climada.sif recipe.txt\"\n", + "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ + "### 3. Configure jupyterhub\n", "\n", - "### Using a virtual environment in a Jupyter notebook\n", + "create a file `~/.config/euler/jupyterhub/jupyterlabrc` with the following content:\n", "\n", - "By default the pre-installed climada version is running in your notebooks. If you want to use climada from source you can simply install a python kernel from the `climada_venv` environment, see [Working with Git branches](#working-with-git-branches)\n", + "```\n", + "#!/bin/bash\n", "\n", - "Install an IPyhton-kernel:\n", + "# Import required modules\n", + "module purge\n", + "module load stack/2024-05 gcc/13.2.0 python/3.11.6_cuda eth_proxy\n", "\n", - "```bash\n", - "source /cluster/project/climate/$USER/climada_venv/bin/activate\n", - "python -m ipykernel install --user --name climada_venv\n", - "```\n", "\n", - "Start a new JupyterLab server, the `climada_venv` kernel should appear in the list of available kernels in JupyterHub." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ + "# Setup the required env\n", + "export JUPYTER_CONFIG_PATH=$HOME/.jupyterlab:$PYTHON_EULER_ROOT/share/jupyter\n", + "export JUPYTER_CONFIG_DIR=$HOME/.jupyterlab\n", + "export JUPYTER_PATH=$PYTHON_EULER_ROOT/share/jupyter\n", + "export JUPYTERLAB_DIR=$PYTHON_EULER_ROOT/share/jupyter/lab\n", + "export JUPYTERLAB_ROOT=$PYTHON_EULER_ROOT\n", "\n", - "## Trouble shooting\n", + "export JUPYTER_HOME=${JUPYTER_HOME:-$HOME}\n", + "export JUPYTER_DIR=${JUPYTER_DIR:-/}\n", + "export JUPYTER_EXTRA_ARGS=${JUPYTER_EXTRA_ARGS:-}\n", "\n", - "### 1. Python Module not found or available\n", + "warn_jupyterhub\n", "\n", - "- Make sure your python environment is activated.\n", - "- Run `pip install --upgrade MISSING_MODULE`.\n", + "sleep 1\n", "\n", - "### 2. Upgrading from Python 3.9 or 3.10\n", + "echo $PYTHON_EULER_ROOT\n", + "echo $JUPYTER_EXTRA_ARGS\n", + "echo $PROXY_PORT\n", "\n", - "Virtual environments created are i.g. only working for the Python version they were created with.\n", - "In particular Python kernels from 3.9 environments will fail to connect in a Jupyter notebook on https://jupyter.euler.hpc.ethz.ch/.\n", + "export PYTHON_ROOT=/opt/software/conda/envs/climada_env\n", + "module purge\n", "\n", - "- It's suggested to create new environments and remove the old kernels from `~/.local/share/jupyter/kernels/`.\n", + "export APPTAINER_BIND=\"/cluster,$TMPDIR,$SCRATCH\"\n", + "singularity exec --nv \\\n", + " --env=\"NVIDIA_VISIBLE_DEVICES=all\" \\\n", + " --bind /cluster/project/[path/to]/climada_python:/opt/climada_workspace/climada_python \\\n", + " --bind /cluster/project/[path/to]/climada_petals:/opt/climada_workspace/climada_petals \\\n", + " /cluster/project/[path/to]/climada.sif \\\n", + " /bin/bash <