Virtual environment magement tool

[offchan42]

1. Can I use other virtual environment management tool instead of conda or venv? For example, I want to try poetry, pdm, pipenv.
2. Is kedro tightly coupled with conda or venv?
3. What is the recommended tool you use internally?

Personally I'm using conda but I find it to be quite messy that I have to maintain both environment.yml and requirements.txt because you might have python version in the conda env and you might have a numpy in requirements.txt. Not sure of the best practices. I have never done a serious virtul environment management before. I always just do pip install latest version of every package on the base environment and share it with every project because dependency conflict rarely happens.

Please offer an opinionated advice based on your virtual env workflow.

[datajoely]

Hi @off99555 - this is a good question and to answer your question I'm going to give an annoying answer and say 'it depends'. I'll also split my answer in two by talking about two separate problems:

1) What should I use to manage my environments? (venv, conda, docker etc.)

You have complete flexibility to do what you want. My personal preference is conda here - someone once showed me this great little PyCon talk discussing the tradeoffs between the various environment managers out there and helped justify why conda works for me. Obligatory XKCD.

2) What should I use to manage my dependencies (pip, conda, poetry, pipenv etc)

This is a bit tricker - Whilst you CAN use whatever you want, bout our docs lay out our recommended approach to managing project level dependencies here. In my experience it's good to worry about dependency conflicts from day 1 as they quickly come to bite you when (a) the number of people working on the same project grows (b) the projects last a long time and need to be maintained (c) you want to use a niche library which has some quirks.

Kedro employs a mechanism of dependency resolution via the commands kedro install and kedro build-reqs, requirements are stated within your requirements.in file and the requirements.txt file functions as your compiled 'lock' file.

This is a process we employ so that we can guarantee that dependencies don't conflict in a mostly automated way. We use pip-tools behind the scenes, pip has recently released their own version of this but it was much slower last time we compared the two side by side.

We have lots of feedback that this is a bit complicated today and have a design which will simplify this in our next major version 0.18.0.