Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch from Poetry to uv #1294

Merged
merged 26 commits into from
Jan 24, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
870154c
Update pyproject.toml metadata format for poetry 2.0 (closer to PEP 621)
ewjoachim Jan 19, 2025
db7bf30
Switch to uv: pyproject.toml & lockfile
ewjoachim Jan 19, 2025
b6d0485
Pre-commit
ewjoachim Jan 19, 2025
3d02cea
Devcontainer
ewjoachim Jan 21, 2025
725f9c1
Documentation & dev env
ewjoachim Jan 21, 2025
3bbcbfd
Remove procrastinate's own version from the lock (needs uv 0.5.21)
ewjoachim Jan 21, 2025
fc2b50f
CI
ewjoachim Jan 21, 2025
0ecc38c
Scripts
ewjoachim Jan 21, 2025
41ccaa3
Readthedocs
ewjoachim Jan 21, 2025
940bf7a
Noxfile
ewjoachim Jan 21, 2025
a31a723
Dockerfile (who uses this ???)
ewjoachim Jan 21, 2025
0dcd081
Tell uv that version may change on every commit
ewjoachim Jan 21, 2025
2d14ea4
We need a recent uv
ewjoachim Jan 21, 2025
68e09c2
Add default tag
ewjoachim Jan 21, 2025
91bd89b
Add . in pythonpath for pytest
ewjoachim Jan 21, 2025
73fe3ee
Try using hatchling instead of setuptools
ewjoachim Jan 21, 2025
191ebc0
Move procrastinate demos under procrastinate
ewjoachim Jan 21, 2025
cdc6c53
Exclude demos from build
ewjoachim Jan 21, 2025
62309cc
Fix docker compose raising warning
ewjoachim Jan 21, 2025
0492add
The option is called --group, not --with-group
ewjoachim Jan 21, 2025
c180f88
Readthedocs needs all extras
ewjoachim Jan 21, 2025
24c6515
pyright broken because pyright python issued a .post0
ewjoachim Jan 21, 2025
29f578f
Improve CI
ewjoachim Jan 22, 2025
488207a
Fix incorrect async pattern in test_shell
ewjoachim Jan 24, 2025
c10fc9f
Fix bootstrap script
ewjoachim Jan 24, 2025
d0470c2
Instanciating the worker means you need an async context, because of …
ewjoachim Jan 24, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .devcontainer/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -13,5 +13,5 @@ RUN sudo apt-get update \

USER vscode

RUN pipx install poetry \
&& poetry completions bash >> ~/.bash_completion
RUN pipx install uv \
&& uv generate-shell-completion bash >> ~/.bash_completion
3 changes: 1 addition & 2 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,7 @@
"PGHOST": "127.0.0.1",
"PGPASSWORD": "password",
"PGUSER": "postgres",
"POETRY_VIRTUALENVS_IN_PROJECT": "true",
"PROCRASTINATE_APP": "procrastinate_demos.demo_async.app.app",
"PROCRASTINATE_APP": "procrastinate.demos.demo_async.app.app",
"VIRTUAL_ENV": "${containerWorkspaceFolder}/.venv"
},
"service": "app",
Expand Down
49 changes: 17 additions & 32 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@ on:
tags:
- "*"

env:
UV_FROZEN: "true"

jobs:
build:
strategy:
Expand Down Expand Up @@ -37,18 +40,13 @@ jobs:
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4

- name: Install poetry
run: pipx install poetry

- uses: actions/setup-python@v5
- name: Install the latest version of uv
uses: astral-sh/setup-uv@b5f58b2abc5763ade55e4e9d0fe52cd1ff7979ca # v5
with:
python-version: "${{ matrix.python-version }}"
cache: "poetry"

- run: poetry install --all-extras
python-version: ${{ matrix.python-version }}

- name: Run tests
run: scripts/tests --cov=procrastinate --cov-branch
run: uv run pytest --cov=procrastinate --cov-branch
env:
COVERAGE_FILE: ".coverage.${{ matrix.python-version }}"
PGHOST: localhost
Expand All @@ -69,7 +67,7 @@ jobs:
- "current_version_without_post_migration"
- "stable_version_without_post_migration"

name: "acceptance-test"
name: "e2e ${{ matrix.mode }}"
runs-on: ubuntu-latest

services:
Expand All @@ -88,13 +86,10 @@ jobs:
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4

- name: Install poetry
run: pipx install poetry

- uses: actions/setup-python@v5
- name: Install the latest version of uv
uses: astral-sh/setup-uv@b5f58b2abc5763ade55e4e9d0fe52cd1ff7979ca # v5
with:
python-version: "3.12"
cache: "poetry"

- name: Get latest tag
id: get-latest-tag
Expand All @@ -103,7 +98,7 @@ jobs:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

- name: Run tests
run: pipx run nox -s ${{ matrix.mode }}
run: uvx nox -s ${{ matrix.mode }}
env:
PGHOST: localhost
PGUSER: postgres
Expand All @@ -116,26 +111,23 @@ jobs:
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4

- name: Install poetry
run: pipx install poetry

- uses: actions/setup-python@v5
- name: Install the latest version of uv
uses: astral-sh/setup-uv@b5f58b2abc5763ade55e4e9d0fe52cd1ff7979ca # v5
with:
python-version: "3.9"
cache: "poetry"

- name: Install dependencies
run: poetry install --all-extras --with=types
run: uv sync --all-extras --group=types

- name: Activate virtualenv
run: echo "$(poetry env info --path)/bin" >> $GITHUB_PATH
run: echo ".venv/bin" >> $GITHUB_PATH

- name: Extract pyright version from pre-commit
id: pre-commit-pyright-version
run: >
yq '.repos
| filter(.repo == "https://github.com/RobertCraigie/pyright-python").0.rev
| "pyright-version="+sub("^v", "")'
| "pyright-version="+sub("^v", "") | sub(".post\d+$"; "")'
.pre-commit-config.yaml >> $GITHUB_OUTPUT

- uses: jakebailey/pyright-action@v2
Expand All @@ -147,6 +139,7 @@ jobs:
runs-on: ubuntu-latest
needs:
- build
- acceptance
- static-typing
steps:
- name: Report success
Expand Down Expand Up @@ -187,15 +180,7 @@ jobs:
- build
- static-typing
steps:
- name: Install poetry
run: |
pipx install poetry
pipx inject poetry 'poetry-dynamic-versioning[plugin]'

- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4

- name: Build wheel and sdist
run: poetry build

- name: Publish package distributions to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
73 changes: 58 additions & 15 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,24 +29,65 @@ repos:
- id: trailing-whitespace
- id: mixed-line-ending
- repo: https://github.com/RobertCraigie/pyright-python
rev: v1.1.391
rev: v1.1.392.post0
hooks:
- id: pyright
additional_dependencies:
- aiopg==1.4.0
- anyio==4.7.0
- alabaster==0.7.16 ; python_full_version < '3.10'
- alabaster==1.0.0 ; python_full_version >= '3.10'
- asgiref==3.8.1
- async-timeout==4.0.3
- attrs==24.3.0
- contextlib2==21.6.0
- babel==2.16.0
- certifi==2024.12.14
- charset-normalizer==3.4.1
- colorama==0.4.6 ; sys_platform == 'win32'
- contextlib2==21.6.0 ; python_full_version < '3.10'
- croniter==6.0.0
- django-stubs==5.1.1
- django==5.1.5
- django==4.2.18 ; python_full_version < '3.10'
- django==5.1.5 ; python_full_version >= '3.10'
- django-stubs==5.1.2
- django-stubs-ext==5.1.2
- docutils==0.21.2
- greenlet==3.1.1 ; (python_full_version < '3.14' and platform_machine ==
'AMD64') or (python_full_version < '3.14' and platform_machine == 'WIN32')
or (python_full_version < '3.14' and platform_machine == 'aarch64') or
(python_full_version < '3.14' and platform_machine == 'amd64') or (python_full_version
< '3.14' and platform_machine == 'ppc64le') or (python_full_version <
'3.14' and platform_machine == 'win32') or (python_full_version < '3.14'
and platform_machine == 'x86_64')
- idna==3.10
- imagesize==1.4.1
- importlib-metadata==8.5.0 ; python_full_version < '3.10'
- jinja2==3.1.5
- markupsafe==3.0.2
- packaging==24.2
- psycopg==3.2.4
- psycopg-pool==3.2.4
- psycopg2-binary==2.9.10
- psycopg[pool]==3.2.3
- pygments==2.19.1
- python-dateutil==2.9.0.post0
- sphinx==7.4.7
- sqlalchemy==2.0.36
- pytz==2024.2
- requests==2.32.3
- six==1.17.0
- snowballstemmer==2.2.0
- sphinx==7.4.7 ; python_full_version < '3.10'
- sphinx==8.1.3 ; python_full_version >= '3.10'
- sphinxcontrib-applehelp==2.0.0
- sphinxcontrib-devhelp==2.0.0
- sphinxcontrib-htmlhelp==2.1.0
- sphinxcontrib-jsmath==1.0.1
- sphinxcontrib-qthelp==2.0.0
- sphinxcontrib-serializinghtml==2.0.0
- sqlalchemy==2.0.37
- sqlparse==0.5.3
- tomli==2.2.1 ; python_full_version < '3.11'
- types-pyyaml==6.0.12.20241230
- typing-extensions==4.12.2
- tzdata==2024.2 ; sys_platform == 'win32'
- urllib3==2.3.0
- zipp==3.21.0 ; python_full_version < '3.10'
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.8.4
hooks:
Expand All @@ -57,11 +98,13 @@ repos:
rev: v1.1.2
hooks:
- id: doc8
- repo: https://github.com/ewjoachim/poetry-to-pre-commit
rev: 2.2.0
hooks:
- id: sync-repos
args: [--map=pyright-python=pyright, --map=ruff-pre-commit=ruff]

- id: sync-hooks-additional-dependencies
args: ['--bind=pyright=main,types']
- repo: local
hooks:
- id: sync-pre-commit
name: Sync pre-commit hooks
language: python
entry: scripts/sync-pre-commit.py
additional_dependencies:
- uv
- ruamel.yaml
4 changes: 2 additions & 2 deletions .readthedocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,9 @@ build:
python: "latest"
jobs:
post_create_environment:
- python -m pip install poetry
- python -m pip install uv
post_install:
- VIRTUAL_ENV=$READTHEDOCS_VIRTUALENV_PATH python -m poetry install --with docs
- UV_PROJECT_ENVIRONMENT=$READTHEDOCS_VIRTUALENV_PATH python -m uv sync --all-extras --group docs

sphinx:
configuration: docs/conf.py
Expand Down
52 changes: 21 additions & 31 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,9 @@ the following assumptions:
- You're using `MacOS` or `Linux`, and `bash` or `zsh`.
- You already have `python3` available
- Either:
- you already have `poetry`, `pre-commit` and `nox` installed
- or you have `pipx` installed and you're ok installing those 3 tools with `pipx`
- or you don't have `pipx` installed but it's ok if we install it for you
- you already have `uv`, `pre-commit` and `nox` installed
- or you have `uv` installed and you're ok installing the 2 other tools with `uv`
- or you don't have `uv` installed but it's ok if we install it for you
- Either:
- you've already setup a PostgreSQL database and environment variables (`PG*`)
are set
Expand Down Expand Up @@ -80,26 +80,27 @@ $ /usr/local/opt/libpq/bin/createdb

### Set up your development environment

The development environment is managed by [poetry]. It's a tool that manages
The development environment is managed by [uv]. It's a tool that manages
dependencies and virtual environments. We also use [pre-commit] to keep the code
clean.
clean and [nox] to run some tests.

If you don't already have `poetry` or `pre-commit` installed, you can
If you don't already have `uv`, `pre-commit` or `nox` installed, you can
install them with:

```console
$ scripts/bootstrap
```

This will install [pipx] if necessary and use it to install `poetry` and
This will install [uv] if necessary and use it to install `nox` and
`pre-commit`.

Then, install Procrastinate with development dependencies in a virtual environment:

```console
$ poetry env use 3.{x} # Select the Python version you want to use (replace {x})
$ poetry install
$ poetry shell # Activate the virtual environment
$ uv venv --python=3.{x} # Select the Python version you want to use (replace {x})
$ uv sync # Install the project and its dependencies
$ uv run $SHELL # Activate the virtual environment
$ exit # Quit the virtual environment
```

You can check that your Python environment is properly activated:
Expand Down Expand Up @@ -397,8 +398,8 @@ When possible, we're trying to avoid duplicating code, with designs such as

## Dependencies management

Dependencies for the package are handled by Poetry in
[`pyproject.toml`](https://github.com/procrastinate-org/procrastinate/blob/main/pyproject.toml#L25).
Dependencies for the package are handled by uv in
[`pyproject.toml`](https://github.com/procrastinate-org/procrastinate/blob/main/pyproject.toml).
Whenever possible, we avoid pinning or putting any kind of limits on the
requirements. We'll typically only do that if we know that there's a
known conflict with specific versions. Typically, even if we support a subset of
Expand All @@ -407,28 +408,17 @@ and if users use procrastinate with unsupported Django version and it works for
them, everyone is happy.

Dependencies for the development environment are kept in
[`poetry.lock`](https://github.com/procrastinate-org/procrastinate/blob/main/poetry.lock).
[`uv.lock`](https://github.com/procrastinate-org/procrastinate/blob/main/uv.lock).
Those are updated regularily by [Renovate](https://docs.renovatebot.com/) which
merges their own PRs.
The versions in `pre-commit-config.yaml` are kept in sync with `poetry.lock`
by the `pre-commit` hook
[poetry-to-pre-commit](https://github.com/procrastinate-org/procrastinate/blob/main/.pre-commit-config.yaml#L61).
The versions in `pre-commit-config.yaml` are kept in sync with `uv.lock`
by a local `pre-commit` hook
[script](https://github.com/procrastinate-org/procrastinate/blob/main/scripts/sync-pre-commit.py).

If you need to recompute the lockfile in your PR, you can use:

```console
$ # Update all the pinned dependencies in pyproject.toml & all versions in poetry.lock
$ # (there are actually no pinned dependencies in pyproject.toml, so this only updates the
$ # lockfile).
$ poetry update

$ # Similarly, update dependencies in the lockfile. In procrastinate, it's equivalent
$ # to the command above
$ poetry lock

$ # Recompute the lockfile (e.g. after the pyproject.toml was updated) without trying
$ # to update anything
$ poetry lock --no-update
$ uv lock
```

## Core contributor additional documentation
Expand All @@ -451,7 +441,7 @@ automated. This works with pre-release too.
When creating the release, GitHub will save the release info and create a tag
with the provided version. The new tag will be seen by GitHub Actions, which
will then create a wheel (using the tag as version number, thanks to
`poetry-dynamic-versioning`), and push it to PyPI (using Trusted publishing).
`versioningit`), and push it to PyPI (using Trusted publishing).
That tag should also trigger a ReadTheDocs build, which will read GitHub
releases (thanks to our `changelog` extension) which will write the changelog
in the published documentation (transformed from `Markdown` to
Expand All @@ -469,8 +459,8 @@ also rebuild the stable and latest doc on [readthedocs](https://readthedocs.org/

[editorconfig]: https://editorconfig.org/
[libpq environment variables]: https://www.postgresql.org/docs/current/libpq-envars.html
[pipx]: https://pipx.pypa.io/stable/
[poetry]: https://python-poetry.org/
[uv]: https://docs.astral.sh/uv
[pre-commit]: https://pre-commit.com/
[Procrastinate releases]: https://github.com/procrastinate-org/procrastinate/releases
[Pytest]: https://docs.pytest.org/en/latest/
[nox]: https://nox.thea.codes/en/stable/
8 changes: 4 additions & 4 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
FROM python:3

RUN pip install poetry
RUN pip install uv

ARG UID=1000
ARG GID=1000
Expand All @@ -10,7 +10,7 @@ USER $UID:$GID
ENV HOME="/src"

COPY pyproject.toml ./
COPY poetry.lock ./
RUN poetry install
ENTRYPOINT ["poetry", "run"]
COPY uv.lock ./
RUN uv sync
ENTRYPOINT ["uv", "run"]
CMD ["procrastinate", "worker"]
6 changes: 3 additions & 3 deletions dev-env
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,9 @@ echo ""
echo "Database is ready!"
echo ""

export PROCRASTINATE_APP=procrastinate_demos.demo_async.app.app
export PROCRASTINATE_APP=procrastinate.demos.demo_async.app.app
export PATH="$(pwd)/scripts/:$PATH"
source $(poetry env info -p)/bin/activate
source .venv/bin/activate

if ! pg_dump --schema-only --table=procrastinate_jobs 1>/dev/null 2>&1; then
echo "Applying migrations"
Expand All @@ -63,4 +63,4 @@ echo "- htmldoc: Opens the locally built sphinx documentation in your browser"
echo "- lint: Run code formatters & linters"
echo "- docs: Build doc"
echo ""
echo 'Quit the poetry shell with the command `deactivate`'
echo 'Quit the shell with the command `deactivate`'
Loading
Loading