From 75868b280eea941c1bef739c5405ff275996451c Mon Sep 17 00:00:00 2001 From: f-PLT Date: Wed, 15 May 2024 15:35:34 -0400 Subject: [PATCH] Update README.md --- README.md | 188 ++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 168 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index b306a21..e2f2693 100644 --- a/README.md +++ b/README.md @@ -5,44 +5,139 @@ This project has only been tested in a Linux (Debian based) environment and assumes some basic tools for development are already installed. -The project uses a Makefile to automate most operations. If you can run make on your -machine there's a good chance this will work on your machine. +The project uses a Makefile to automate most operations. If make is available on your +machine there's a good chance this will work. -Please consult the [Makefile](Makefile) before trying to use it. +The following Makefile files should not be modified, but can be consulted: -## Basic Information +* [Makefile](Makefile) : Orchestration of the different files +* [base.make](.make/base.make) : Shared utilities, project agnostic. + +The following Makefile files are project or user specific: -Much of this repository is organized through the Makefile. +* [Makefile.variables](Makefile.variables) : Shared project variables. +* [Makefile.targets](Makefile.targets) : Shared project targets. +* [Makefile.private](Makefile.private.example) : User specific variables and targets. + +## Basic Information The different targets and their description can be examined by executing the command `make targets` -![](images/make_targets.png) +![](img/make_targets.png) ## Installation -This project assumes environment management will be done with Conda. It would, however, -be possible to create you own environment with, for example, poetry, pyenv or virtualenv. +This project assumes environment management will be done with `Conda` or directly through +`Poetry`. + +* [Poetry](https://python-poetry.org/docs/basic-usage/) +* [Conda](https://conda.io/projects/conda/en/latest/user-guide/getting-started.html) -**Do note that this project is configured to use [Poetry](https://python-poetry.org/) -and as such, this tool is required to install it.** +While it is possible to manage the environment with, for example, pyenv or virtualenv, +those specific use cases are not supported by the Makefile and require users to set up +their own environments beforehand. -Poetry is not included in the [environment.yml](environment.yml), due to some problems +For detailed information about `Poetry` and `Conda`: + +If you want to use something else than `Conda` or `Poetry` to manage environment isolation, +it is recommended to follow +[Poetry's guidelines on managing environments](https://python-poetry.org/docs/managing-environments/) + +Poetry is not included in the [environment.yml](environment.yml), due to some possible problems in compute cluster environments, but will be installed automatically if needed by most `install` targets. +Currently, the project runs on Python version 3.10. + +### Environment management choices + +Environment management can become quite complicated. Using Conda allows a certain +ease of management since the Poetry installation is contained inside the created Conda +environment. + +However, some computing environments do not permit the use of Conda (like certain SLURM +clusters). This is why the `pipx` option for Poetry is also enabled in this project. + +**Unless you really know what you are doing, it is not recommended to install Poetry +as a standalone tool (with pipx) while also using Conda environments.** + +### Conda targets + If you need or want to install Conda: ``` -make install-conda +make conda-install ``` To create the conda environment: ``` -make create-env +make conda-create-env +``` + +To remove the conda environment: +``` +make conda-clean-env +``` + +Make sure to activate the configured environment before installing this package. + +### Poetry targets + +There are 2 possibilities available in how to manage `Poetry` if it is not already +configured on your system. + +**It is not recommended to install `Poetry` in the same environment that will be managed +by `Poetry`; It should be installed by `Conda` (as in, `conda install poetry`) or +on the system using `pipx`, in order to minimize dependency conflicts.** + +The following target will first try to install `Poetry` in the active `Conda` +environment; if it can't find `Conda`, it will proceed to install via `pipx` + +``` +make poetry-install-auto +``` + +To install `Poetry` using `Conda`: + +``` +make conda-poetry-install +``` + +Using `pipx` will instead allow environment management directly with `Poetry`. The +following target will also make `Poetry` use a Python 3.10 environment for this +project. + +``` +make poetry-install-pipx +``` + +This will also create a virtual environment managed by `Poetry`. + +A standalone environment can also be created later using the `make poetry-create-env` +command, and removed with the `make poetry-remove-env` command. + +Information about the currently active environment used by Poetry, +whether Conda or Poetry, can be seen using the `make poetry-env-info` command. + +Both install methods can also be cleaned up: + +``` +make conda-poetry-uninstall +# or +make poetry-uninstall-pipx ``` -After you have created your Conda environment, or you want to manage your environment -yourself, make sure to activate it properly before installing this package. +**Important note!** + +If you have an active `Conda` environment and install `Poetry` using `pipx`, +you will have to use `poetry run python ` instead of +`python `, (which is normal when using Poetry) as +`python` will use Conda's active environment. + +### Install targets + +**All `install` targets will first check if `Poetry` is available and try to install +it with the `make poetry-install-auto` target.** To install the package, development dependencies and CLI tools (if available): ``` @@ -54,6 +149,52 @@ To install only the package, without development tools: make install-package ``` +## First time user quick setup + +### Conda +The easiest and quickest way to get up and running with Conda. + +Create Conda environment (will check for Conda and install it if not found): + +``` +make conda-create-env +``` + +Activate Conda environment (substitute with your if something else +than `conda`: + +``` +conda activate lab-project-template +``` + +Install package: + +``` +make install +``` + +### Poetry + +The easiest and quickest way to get up and running with Poetry. + +Install pipx and Poetry and activate project environment : + +``` +make poetry-install +``` + +**Or, if Poetry is already available:** + +``` +make poetry-create-env +``` + +Install package: + +``` +make install +``` + ## Basic automations To run linting checks with `flake8`, `pylint`, `black` and `isort`: @@ -76,17 +217,24 @@ To run tests: make test ``` -To run all tests, including the integration test: -``` -make test-all -``` ## Data +## Experiment tracking + +Nothing is set up for now, but since Weights and Bias is accessible to MILA and DRAC, it +will probably be the way to go. + + +## Training + ## Contributing to this repository See [Contributing guidelines](CONTRIBUTING.md) -## Configurations + +### Configurations Configurations are in the [config/](config) folder. + +### Tests