Discussion: Using Poetry for dependency management

[kushaldas]

Poetry tool

Poetry is a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you.

• Basic usage

In an already existing project, you can run poetry init to create the pyproject.toml file and either choose the interactive mode, or update the file directly with the dependencies.

poetry install --no-root

This command will install all the dependencies and will also create the poetry.lock file.

How to upgrade the dependencies?

One can choose the specific dependencies to upgrade.

poetry update requests redis

To update all the dependencies together:

poetry update

How do other projects manage dependencies?

• Onionshare cli, except flask-socketio = "5.0.1" every thing is marked as *, means always latest package.
• isort Only dependency hardcoded for minor revisions, all dev-dependencies for minor revisions too.
• noggin also hardcodes for minor revisions.

Related Python PEP

https://www.python.org/dev/peps/pep-0631/

Major work required to move

• All of our scripts are based on requirements.txt file style, so we will have to update those.
• Our final Debian package building is still dependent on requirements.txt file format and pip to install the dependencies during Python package build.

[conorsch]

Neat write-up, thanks for sharing your early thoughts, @kushaldas.

Our final Debian package building is still dependent on requirements.txt file format

There's a poetry export option that emits a requirements.txt with pinned hashes, based on the poetry.lock file. Maybe compare that to the requirements.txt files we currently have. For example, I expect they won't include things like build tooling via pip-tools's --allow-unsafe option. It's also not clear to me how the export subcommand interacts with "dev" or "prod" dependencies for poetry.

[sssoleileraaa]

First impressions

Couple highlights for me when looking into poetry:

• The docs are useful and easy to understand: https://python-poetry.org/docs/, https://python-poetry.org/docs/dependency-specification/, https://python-poetry.org/docs/cli/, etc
• The cli is easy to use (see example session below)
• Seems kind of messy to mix config for black and isort with our dependency config, i.e. everything goes into the same pyproject.toml file
• The --dev-only feature has not been released (waiting for python-poetry/poetry@d2485b8 to go into 1.2.0 but it will provide an easy way to install dev-only dependencies (for static analysis tools and CI scripts that don't require application code to run) using poetry install --dev-only. poetry install --no-dev installs only prod dependencies and will even remove any installed dependencies that are not prod. the reverse also works: poetry install --no-dev followed by poetry install --dev-only will remove any installed dependencies that are not dev-only.
  • I think this is an important feature to wait for, if we decide to use this tool
• Updating a single package is easy, e.g. pyqt 5.11.3 -> 5.15.4:

  (securedrop-client-p8SLegPY-py3.7) user@dev:securedrop-client(use-poetry⚡) » git diff

  diff --git a/pyproject.toml b/pyproject.toml
  index c63c051..2217634 100644
  --- a/pyproject.toml
  +++ b/pyproject.toml
  @@ -33,7 +33,7 @@ securedrop-sdk = "^0.3.0"
   six = "^1.11.0"
   SQLAlchemy = "^1.3.3"
   urllib3 = "^1.25.10"
  -PyQt5 = "~5.11.3"
  +PyQt5 = "^5.11.3"
   requests = "^2.26.0"
   
   [tool.poetry.dev-dependencies]
  @@ -58,7 +58,7 @@ py = "^1.10.0"
   PyAutoGUI = "^0.9.48"
   pycodestyle = "^2.4.0"
   pyflakes = "^2.0.0"
  -PyQt5 = "~5.11.3"
  +PyQt5 = "^5.11.3"
   pytest = "^5.2.1"
   pytest-cov = "^2.8.1"
   pytest-mock = "^1.10.0"

  (securedrop-client-p8SLegPY-py3.7) user@dev:securedrop-client(use-poetry⚡) » poetry update pyqt5
  Updating dependencies
  Resolving dependencies... (2.0s)
  
  Writing lock file
  
  Package operations: 2 installs, 1 update, 0 removals
  
    • Installing pyqt5-qt5 (5.15.2)
    • Installing pyqt5-sip (12.9.0)
    • Updating pyqt5 (5.11.3 -> 5.15.4)

• Updating a single package back to an earlier version with a new dependency specification is easy (using ~ instead of ^ for instance -- to learn more, see https://python-poetry.org/docs/dependency-specification/):

  (securedrop-client-p8SLegPY-py3.7) user@dev:securedrop-client(use-poetry⚡) » #switch back to `~` in the pyproject.toml file
  (securedrop-client-p8SLegPY-py3.7) user@dev:securedrop-client(use-poetry⚡) » poetry update pyqt5
  poetry update pyqt5
  Updating dependencies
  Resolving dependencies... (2.2s)
  
  Writing lock file
  
  Package operations: 0 installs, 1 update, 2 removals
  
    • Removing pyqt5-qt5 (5.15.2)
    • Removing pyqt5-sip (12.9.0)
    • Updating pyqt5 (5.15.4 -> 5.11.3)

Example session

user@dev:securedrop-client(use-poetry⚡) » poetry --version
Poetry version 1.1.7
user@dev:securedrop-client(use-poetry⚡) » poetry self update 
You are using the latest version
user@dev:securedrop-client(use-poetry⚡) » poetry shell
Spawning shell within /home/user/.cache/pypoetry/virtualenvs/securedrop-client-p8SLegPY-py3.7
user@dev:securedrop-client(use-poetry⚡) » . /home/user/.cache/pypoetry/virtualenvs/securedrop-client-p8SLegPY-py3.7/bin/activate
(securedrop-client-p8SLegPY-py3.7) user@dev:securedrop-client(use-poetry⚡) » poetry install --no-dev
Updating dependencies
Resolving dependencies... (48.9s)

Writing lock file

Package operations: 20 installs, 0 updates, 0 removals

  • Installing typing-extensions (3.10.0.0)
  • Installing zipp (3.5.0)
  • Installing certifi (2018.11.29)
  • Installing charset-normalizer (2.0.4)
  • Installing greenlet (1.1.1)
  • Installing idna (2.10)
  • Installing importlib-metadata (4.6.4)
  • Installing markupsafe (1.1.1)
  • Installing six (1.16.0)
  • Installing urllib3 (1.26.6)
  • Installing mako (1.1.4)
  • Installing python-dateutil (2.8.2)
  • Installing python-editor (1.0.4)
  • Installing requests (2.26.0)
  • Installing sqlalchemy (1.4.22)
  • Installing alembic (1.6.5)
  • Installing arrow (0.12.1)
  • Installing chardet (3.0.4)
  • Installing pathlib2 (2.3.6)
  • Installing securedrop-sdk (0.3.0)

Installing the current project: securedrop-client (0.5.0)
(securedrop-client-p8SLegPY-py3.7) user@dev:securedrop-client(use-poetry⚡) » poetry env info     

Virtualenv
Python:         3.7.3
Implementation: CPython
Path:           /home/user/.cache/pypoetry/virtualenvs/securedrop-client-p8SLegPY-py3.7
Valid:          True

System
Platform: linux
OS:       posix
Python:   /usr
(securedrop-client-p8SLegPY-py3.7) user@dev:securedrop-client(use-poetry⚡) » poetry build
Building securedrop-client (0.5.0)
- Building sdist
- Built securedrop-client-0.5.0.tar.gz
- Building wheel
- Built securedrop_client-0.5.0-py3-none-any.whl
(securedrop-client-p8SLegPY-py3.7) user@dev:securedrop-client(use-poetry⚡) » exit
user@dev:securedrop-client(use-poetry⚡) »

[sssoleileraaa]

I forgot to mention that you can check out my branch to test poetry with the client (I think it makes sense to use https://python-poetry.org/docs/dependency-specification/#caret-requirements for prod requirements (minus pyqt5 unfortunately since that version needs tighter restrictions due to using system qt) and https://python-poetry.org/docs/dependency-specification/#wildcard-requirements for dev-only dependencies to use the latest (by default).

I really like poetry and would be happy to make more of an effort to get it working with our in-house packaging and CI scripts (which will take some effort as @kushaldas mentioned in the issue description), but, again, I think it makes the most sense to wait until the --dev-only install feature in the next release before we start migrating our scripts. I could also see us getting started early and moving to an in-between state where generate the requirements.txt file as @conorsch mentioned. We'll have to see what to do about generating the dev-requirements.txt file as well. Since we manually maintain https://github.com/freedomofpress/securedrop-client/blob/use-poetry/requirements/build-requirements.txt and only pin to our own local wheels there, we can just keep doing that for now, but also something to think about generating using poetry.

[sssoleileraaa]

I could also see us getting started early and moving to an in-between state where generate the requirements.txt file as @conorsch mentioned. We'll have to see what to do about generating the dev-requirements.txt file as well.

Looks like we can do the following to generate the desired requirement files:

poetry install && poetry export -f requirements.txt --output requiremetns/dev-requirements.txt
poetry install --no-dev  && poetry export -f requirements.txt --output requiremetns/requirements.txt

[conorsch]

All in all, there's developing consensus that poetry is a sufficiently ergonomic tool that we should consider adopting it. However, during group discussion, @zenmonkeykstop pointed out that there's a show-stopping security bug in the current prod release of poetry, and in fact it's been present for years. The bug means that the hashes in the lockfile are not actually checked: python-poetry/poetry#2422 That's egregious enough that we'll shelve evaluation for a while, at minimum until that issue is resolved in a prod release.

[sssoleileraaa]

-- the --dev-only feature and the bug fix for ^ will go into 1.2.0

Since we only did a quick scan for blocking issues, I think we should do a more thorough scan once 1.2.0 is released to see if anything else remains a blocker.

[legoktm]

The latest poetry version now properly checks hashes (according to the linked bug report).

[gonzalo-bulnes]

Given the above, I'm sold. (If, of course, package hashes are properly checked, which I suppose we can find way to test... 💭).

[eloquence]

As an example of a simple poetry integration for one of our existing repos, see freedomofpress/redmine_docs#48.

[legoktm]

We've since adopted poetry for dependency management.