Implement Poetry build system (mainly so we don't have to commit dynamically generated front end bundles to git) #2725

mquinnfd · 2024-05-24T09:16:34Z

Switching to the Poetry build system

Tip

This PR was worked on iteratively in Draft form, so there's a bit of history in the comments here.

This is being suggested for integration to make building and bundling the front end a bit easier, comments welcome particularly if you have strong opinions on poetry

In a Nutshell

This PR changes the build system used to package Locust over to Poetry.

The main use-case for this currently, was to allow the removal of the (snazzy new) dynamically generated front end bundle from the actual source code, and just let the build process create these for the package during the build.

Caveat

Details

I've tried to keep the package output and metadata like-for-like, because I havn't looked behind the curtain at the Locust release process, CI runners, etc. - it's likely this may need to change for that purpose.

There's plenty I am blissfully unaware of when it comes to maintaining this project, this is just a jumping off point for the sake of conversation - I have no particular love of Poetry, though I do use this day-to-day for projects currently. Alternatives noted were PDM (and now Rye by "the Ruff people") but I have no idea if these support the king of things we're doing here currently.

What

Adds the Poetry build system to the Locust package, this changes the build backend to Poetry's PEP-517 compliant build back-end; Poetry Core

Added Poetry build system
Don't check in Poetry lock file, as we're often used as a library (worth discussion) - but do include it in generated builds, for provenance
Poetry plugin poetry-dynamic-versioning does the job of setuptools-scm to keep the version number in line with VCS info

Why

Discussed with @cyberw here: #2405 (comment)

Allows for pre-build script, which allows us to build the front end on-demand and remove it from VCS
While Poetry is far from perfect, it is at least well maintained by a large community
Package size reduced by 10% due to not packaging tests etc. in the wheel (were these necessary somehow? library users maybe?)

Try it Out

Local dev installation

Details

make install

Installs project dependencies
Installs editable local package of `locust`

Packaging

Details

make build

Installs `poetry` and the required plugin
Builds the project, including the web UI
Packages this, and homogenises the wheel package name

TODO

I wanted to put there out there for discussion before doing things like:

Rework the Github actions workflows for the CI build
Adding documentation and updating the README
Adding improvements to the build process

- Modify pyproject.toml file for poetry build system - Adds build script to generate front end as part of poetry build - Adds renaming script to remove architecture info from wheel file name - Updates Makefile to call the correct build steps with `make build`

mquinnfd · 2024-05-24T09:18:01Z

For the consideration of @cyberw and community, apologies on the delay, sometimes life happens 😓

Makefile

pyproject.toml

mquinnfd · 2024-05-24T10:20:43Z

I'm also unsure if you have squash-on-merge turned on for the repo, just an FYI as I have not lovingly rebased my commits currently

cyberw · 2024-05-24T14:28:39Z

First run of make install failed for me. Probably the shell doesnt update its list of installed things in the middle of a && (could be enough to move poetry self add... to a new line in the Makefile)

...
Using cached jaraco.classes-3.4.0-py3-none-any.whl (6.8 kB)
Installing collected packages: jaraco.classes, cleo, build, keyring, cachecontrol, poetry-plugin-export, poetry
Successfully installed build-1.2.1 cachecontrol-0.14.0 cleo-2.1.0 jaraco.classes-3.4.0 keyring-24.3.1 poetry-1.8.3 poetry-plugin-export-1.8.0
/bin/sh: poetry: command not found
make: *** [setup_build] Error 127

cyberw · 2024-05-24T14:39:58Z

Perhaps it makes more sense to just assume poetry is installed (possibly giving a slightly nicer error message if it is not)

cyberw · 2024-05-24T14:55:23Z

Make build fails on yarn not being installed too. Bonus points if we can give a nice error message or even provide an action to install that.

cyberw · 2024-05-24T14:56:14Z

Make install doesnt actually add locust to my path. Do i need to do something more to set up poetry correctly? (python3 -m locust ... works though)

cyberw · 2024-05-24T15:26:46Z

I think the versioning is only almost right.

python -m locust -V
locust 2.28.0 from /Users/lars/git/locust/locust (Python 3.12.2, OpenSSL 3.3.0)

On an un-tagged master it should be something like locust-2.28.1.dev5 (this version is needed to make good versions for pre-releases that are made whenever something is merged to master, see https://pypi.org/project/locust/#history)

mquinnfd · 2024-05-25T11:28:49Z

First run of make install failed for me. Probably the shell doesnt update its list of installed things in the middle of a && (could be enough to move poetry self add... to a new line in the Makefile)
...
Using cached jaraco.classes-3.4.0-py3-none-any.whl (6.8 kB)
Installing collected packages: jaraco.classes, cleo, build, keyring, cachecontrol, poetry-plugin-export, poetry
Successfully installed build-1.2.1 cachecontrol-0.14.0 cleo-2.1.0 jaraco.classes-3.4.0 keyring-24.3.1 poetry-1.8.3 poetry-plugin-export-1.8.0
/bin/sh: poetry: command not found
make: *** [setup_build] Error 127

Great shout! I ran this again in a clean venv and fixed with 3b55405

mquinnfd · 2024-05-25T11:41:13Z

Make build fails on yarn not being installed too. Bonus points if we can give a nice error message or even provide an action to install that.

Yeah good idea, I don't think there's a reliable cross-platform way to do this really - I've added 9c90aec which gives the following behaviour when you don't have the yarn binary available:

<username>@<hostname> locust-fork % make build
Locust requires the yarn binary to be available in this shell to build the web front-end.
See: https://classic.yarnpkg.com/lang/en/docs/install
make: *** [check-yarn] Error 1

(I assume we're using classic yarn)

We could certainly do something like attempting to install it via NPM after checking if NPM is there?

If npm not installed, echo error -> exit
If yarn not installed, npm install yarn etc. -> continue
If poetry not installed, pip install poetry etc. -> continue
(build happens)

Same goes with poetry - whatever is easier for people, could just warn out to the terminal alone.

mquinnfd · 2024-05-25T11:55:39Z

Make install doesnt actually add locust to my path. Do i need to do something more to set up poetry correctly? (python3 -m locust ... works though)

Would you mind trying again on a clean venv if possible? (with the branch up to date)
I added 7e290ff which, for me, results in:

make install
locust is on the path, and can be called

mquinnfd · 2024-05-25T12:02:42Z

I think the versioning is only almost right.
python -m locust -V
locust 2.28.0 from /Users/lars/git/locust/locust (Python 3.12.2, OpenSSL 3.3.0)
On an un-tagged master it should be something like locust-2.28.1.dev5 (this version is needed to make good versions for pre-releases that are made whenever something is merged to master, see https://pypi.org/project/locust/#history)

I'll need to take a look at this behaviour - it could be that I've made a load of tags testing this and something's getting a bit confused, I can try on a clean branch maybe - it should reflect your latest local git tag though right? What is that, in your case?

user@computer % python -m locust -V
locust 2.28.0.dev0 from /Users/user/dev/locust-fork/locust (Python 3.9.16, OpenSSL 1.1.1v)
user@computer % git describe --tags --abbrev=0
2.28.0.dev0

cyberw · 2024-05-25T12:51:22Z

Make build fails on yarn not being installed too. Bonus points if we can give a nice error message or even provide an action to install that.

Yeah good idea, I don't think there's a reliable cross-platform way to do this really - I've added 9c90aec which gives the following behaviour when you don't have the yarn binary available:

I think this is a reasonable compromise. People may want to install node and yarn in various ways so automating that is out of scope I think.

cyberw · 2024-06-23T07:37:03Z

Makefile

+.SILENT:
+.PHONY: check-poetry
+check-poetry:
+	command -v poetry >/dev/null 2>&1 || { echo >&2 "Locust requires the poetry binary to be available in this shell to build the Python package.\nSee: https://docs.locust.io/en/stable/developing-locust.html#install-locust-for-development"; exit 1; }


I may have found a nicer way to do this:
which poetry > /dev/null || {...
Or maybe that is linux/unix-specific?

Yeah I believe which is unix-specific - the windows equivalent would be where

Come to think of it, I think command itself is POSIX compliant and I currently have no idea if this works on Windows either 😓

Cross platform options likely boil down to:

write a couple more lines (have cases for each platform)

call out to the system's Python and have that check (we need Python anyway right)

maybe just call the binary and catch the errors to the pipe anyway

locust/contrib/fasthttp.py

pyproject.toml

cyberw · 2024-06-23T08:04:45Z

Looking really good now.

As for testing twine/publishing, its ok to break the publishing for a little while (maybe time the merge for when you know you have a couple hours to fix any issues that might arise?). I do have a pypy test account where we could publish, but meh.

I did have an issue with the docker image. Can you try it and see if the same thing happened to you?

> docker build -t locustio/locust:latest .
...
> docker run -p 8089:8089 -v $PWD:/mnt/locust locustio/locust -f /mnt/locust/locustfile.py
[2024-06-23 07:59:40,884] 474c796dc630/INFO/locust.main: Starting web interface at http://0.0.0.0:8089
[2024-06-23 07:59:40,890] 474c796dc630/INFO/locust.main: Starting Locust 2.29.0.dev110
(and then I opened the start page)
Traceback (most recent call last):
  File "/opt/venv/lib/python3.11/site-packages/flask/app.py", line 880, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/flask/app.py", line 865, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/locust/web.py", line 565, in wrapper
    return view_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/locust/web.py", line 170, in index
    return render_template("index.html", template_args=self.template_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/flask/templating.py", line 149, in render_template
    template = app.jinja_env.get_or_select_template(template_name_or_list)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/jinja2/environment.py", line 1084, in get_or_select_template
    return self.get_template(template_name_or_list, parent, globals)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/jinja2/environment.py", line 1013, in get_template
    return self._load_template(name, globals)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/jinja2/environment.py", line 972, in _load_template
    template = self.loader.load(self, name, self.make_globals(globals))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/jinja2/loaders.py", line 126, in load
    source, filename, uptodate = self.get_source(environment, name)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/flask/templating.py", line 65, in get_source
    return self._get_source_fast(environment, template)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/flask/templating.py", line 99, in _get_source_fast
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: index.html

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.11/site-packages/gevent/pywsgi.py", line 1107, in handle_one_response
    self.run_application()
  File "/opt/venv/lib/python3.11/site-packages/gevent/pywsgi.py", line 1053, in run_application
    self.result = self.application(self.environ, self.start_response)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/flask/app.py", line 1498, in __call__
    return self.wsgi_app(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/flask/app.py", line 1476, in wsgi_app
    response = self.handle_exception(e)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/flask_cors/extension.py", line 178, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
                                                ^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/flask/app.py", line 1473, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/flask/app.py", line 882, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/flask_cors/extension.py", line 178, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
                                                ^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/flask/app.py", line 772, in handle_user_exception
    return self.ensure_sync(handler)(e)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/locust/web.py", line 155, in handle_exception
    f"UI got request for {request.path}, but it resulted in a {error.code}: {error.name}",
                                                               ^^^^^^^^^^
AttributeError: 'TemplateNotFound' object has no attribute 'code'
2024-06-23T07:59:44Z {'REMOTE_ADDR': '::ffff:172.17.0.1', 'REMOTE_PORT': '54400', 'HTTP_HOST': '0.0.0.0:8089', (hidden keys: 25)} failed with AttributeError

mquinnfd · 2024-06-23T11:41:39Z

Looking really good now.

As for testing twine/publishing, its ok to break the publishing for a little while (maybe time the merge for when you know you have a couple hours to fix any issues that might arise?). I do have a pypy test account where we could publish, but meh.

...

Yeah sounds good to me I can set aside some time for this 👍
Might end up with some good suggestions for optimising the flow etc. if we make the PR for review at some point as well

I'll give the Dockerfile another whirl now

mquinnfd · 2024-06-23T12:12:45Z

Ok the docker image works now, thanks! I actually put the prebuilt web UI in the wrong location originally.

Updated version should work for you too. (sorry http://example.com whoever you are)

cyberw · 2024-06-23T13:59:57Z

Great stuff! I’ll let @andrewbaldwin44 have a whirl when he gets back on Tuesday. Is there something left to do before you feel it is ready to merge? If not, you can take it out of draft and once Andrew has reviewed we’ll find a good time to merge.

locust/webui/.yarnrc

pyproject.toml

.github/workflows/tests.yml

andrewbaldwin44 · 2024-06-25T13:36:45Z

I noticed when running the tests python -m poetry run tox -e fail_fast_test_main that it runs the build for the frontend as well. If it is only running the tests in locust/test/test_main.py, the build wouldn't be necessary. Maybe we could use SKIP_PRE_BUILD?

andrewbaldwin44 · 2024-06-25T13:37:55Z

@mquinnfd I left a couple of small comments but overall this looks awesome :) Thanks so much for working on this and battling with Windows! 🥇

mquinnfd · 2024-06-26T11:11:14Z

@mquinnfd I left a couple of small comments but overall this looks awesome :) Thanks so much for working on this and battling with Windows! 🥇

Thanks for the review man 🙇
I'll get these bits rounded up this week and then yeah might be time to open it up to the gallery, I'm sure there are some opinionated poetry bits and bobs that could be changed here

mquinnfd · 2024-06-30T12:15:55Z

I noticed when running the tests python -m poetry run tox -e fail_fast_test_main that it runs the build for the frontend as well. If it is only running the tests in locust/test/test_main.py, the build wouldn't be necessary. Maybe we could use SKIP_PRE_BUILD?

Hey this is a great call! This shaves some time off the Windows and Linux builds 👍
Should see them skipping now - like below

Installing the current project: locust (2.29.1.dev122)
Preparing build environment with build-system requirements poetry-core>=1.0.0, poetry-dynamic-versioning>=1.0.0,<2.0.0
Skipping front end build...
fail_fast_test_main: commands[1]> poetry run python -m unittest -f locust.test.test_main

from tmp7ef0_06q_locustfile import TestUser1

.........ssssss.s...s.sss.......ss..s.....ss......sssss.
----------------------------------------------------------------------
Ran 56 tests in 135.491s

OK (skipped=21)
  fail_fast_test_main: OK (152.48=setup[1.67]+cmd[11.98,138.83] seconds)
  congratulations :) (152.62 seconds)

mquinnfd · 2024-07-02T09:25:58Z

Going to open this for Public review and I believe the rest of these comments have been worked through - I'll take the opportunity while people eyeball it, to look at the publish steps next chance I get

mquinnfd added 8 commits May 24, 2024 08:51

remove generated front end bundle

1e23586

add poetry and front end dist to vcs ignore

5d57ba7

chmod on release renaming script

ad90bb6

re-add wheel extension to changed file name

4d58026

ensure vsc autoversioning plugin

6b46420

ensure vcs auto versioning plugin

cf9c05f

revert vscode settings

e5dde14

mquinnfd added 3 commits May 24, 2024 10:28

add local dev install command

fe2b756

add version file to package output

14dcb28

only include poetry lock in sdist

d3d5e81

mquinnfd commented May 24, 2024

View reviewed changes

Makefile Show resolved Hide resolved

mquinnfd commented May 24, 2024

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

mquinnfd commented May 24, 2024

View reviewed changes

pyproject.toml Show resolved Hide resolved

run build setup as one shell

3b55405

check if yarn is installed before building

9c90aec

mquinnfd added 2 commits May 25, 2024 12:46

rename foo binary to yarn

1f72660

deny poetry venvs creation

7e290ff

cyberw reviewed Jun 23, 2024

View reviewed changes

locust/contrib/fasthttp.py Show resolved Hide resolved

cyberw reviewed Jun 23, 2024

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

mquinnfd added 3 commits June 23, 2024 12:19

revert vs code settings

f2a8c34

remove oneshell directive from old installation process

39cf84a

Update maintainers and authors

90ffcc8

move web ui to correct location

ba58f38

andrewbaldwin44 reviewed Jun 25, 2024

View reviewed changes

locust/webui/.yarnrc Outdated Show resolved Hide resolved

pyproject.toml Outdated Show resolved Hide resolved

pyproject.toml Outdated Show resolved Hide resolved

andrewbaldwin44 reviewed Jun 25, 2024

View reviewed changes

.github/workflows/tests.yml Show resolved Hide resolved

merge master

70f240b

cyberw mentioned this pull request Jun 27, 2024

update github actions to get rid of deprecation warnings #2778

Closed

mquinnfd added 2 commits June 28, 2024 11:26

update file exclusions

9369440

skip fe build in fail-fast test

a0f2e2b

mquinnfd marked this pull request as ready for review July 2, 2024 09:25

mquinnfd added 4 commits July 4, 2024 13:10

Merge branch 'master' into poetry-build-system

89da571

pin some version dependencies

f326bdb

update tox version

0259c89

add wheel renaming to prepublishing steps

c9a1072

cyberw merged commit 0da8fa1 into locustio:master Jul 5, 2024
14 checks passed

cyberw changed the title ~~Poetry build system~~ Implement Poetry build system (mainly so we don't have to commit dynamically generated front end bundles to git) Jul 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Poetry build system (mainly so we don't have to commit dynamically generated front end bundles to git) #2725

Implement Poetry build system (mainly so we don't have to commit dynamically generated front end bundles to git) #2725

mquinnfd commented May 24, 2024 •

edited

Loading

mquinnfd commented May 24, 2024

mquinnfd commented May 24, 2024

cyberw commented May 24, 2024

cyberw commented May 24, 2024

cyberw commented May 24, 2024

cyberw commented May 24, 2024

cyberw commented May 24, 2024 •

edited

Loading

mquinnfd commented May 25, 2024

mquinnfd commented May 25, 2024 •

edited

Loading

mquinnfd commented May 25, 2024 •

edited

Loading

mquinnfd commented May 25, 2024

cyberw commented May 25, 2024

cyberw Jun 23, 2024 •

edited

Loading

mquinnfd Jun 23, 2024

cyberw commented Jun 23, 2024

mquinnfd commented Jun 23, 2024 •

edited

Loading

mquinnfd commented Jun 23, 2024

cyberw commented Jun 23, 2024

andrewbaldwin44 commented Jun 25, 2024

andrewbaldwin44 commented Jun 25, 2024

mquinnfd commented Jun 26, 2024

mquinnfd commented Jun 30, 2024

mquinnfd commented Jul 2, 2024

Implement Poetry build system (mainly so we don't have to commit dynamically generated front end bundles to git) #2725

Implement Poetry build system (mainly so we don't have to commit dynamically generated front end bundles to git) #2725

Conversation

mquinnfd commented May 24, 2024 • edited Loading

Switching to the Poetry build system

In a Nutshell

Caveat

What

Why

Try it Out

TODO

mquinnfd commented May 24, 2024

mquinnfd commented May 24, 2024

cyberw commented May 24, 2024

cyberw commented May 24, 2024

cyberw commented May 24, 2024

cyberw commented May 24, 2024

cyberw commented May 24, 2024 • edited Loading

mquinnfd commented May 25, 2024

mquinnfd commented May 25, 2024 • edited Loading

mquinnfd commented May 25, 2024 • edited Loading

mquinnfd commented May 25, 2024

cyberw commented May 25, 2024

cyberw Jun 23, 2024 • edited Loading

Choose a reason for hiding this comment

mquinnfd Jun 23, 2024

Choose a reason for hiding this comment

cyberw commented Jun 23, 2024

mquinnfd commented Jun 23, 2024 • edited Loading

mquinnfd commented Jun 23, 2024

cyberw commented Jun 23, 2024

andrewbaldwin44 commented Jun 25, 2024

andrewbaldwin44 commented Jun 25, 2024

mquinnfd commented Jun 26, 2024

mquinnfd commented Jun 30, 2024

mquinnfd commented Jul 2, 2024

mquinnfd commented May 24, 2024 •

edited

Loading

cyberw commented May 24, 2024 •

edited

Loading

mquinnfd commented May 25, 2024 •

edited

Loading

mquinnfd commented May 25, 2024 •

edited

Loading

cyberw Jun 23, 2024 •

edited

Loading

mquinnfd commented Jun 23, 2024 •

edited

Loading