Pyodide Conversion: fix requirements handling #6859
Conversation
…xt and custom location wheels). enable packaging resources used by the python app
flxmr
changed the title
flxmr
changed the title
|
This is incredible, thanks for diving into this so deeply @flxmr! I'll have to dig into this a little bit to unpack it and I'm out until next week but would like to target this for the 1.5 release in early to mid June. |
| }); | ||
| } | ||
| self.postMessage({type: 'status', msg: `Installing packages`}); | ||
| // a finegrained approach installing dependencies one after another with status updates was implemented previously |
There was a problem hiding this comment.
Unfortunate but better to handle this correctly. Would be great if micropip supported hooks to report progress.
There was a problem hiding this comment.
ah, that was not commented properly for the review! The problem is that the dependency resolution fails for locally installed wheels if they are not in the install command (...) - probably this should be fixed in micropip. I will try to do some tests too and poke the micropip-people a bit in the meantime.
There was a problem hiding this comment.
Sweet thanks! Am trying to test this PR and then get it merged in time for the Panel 1.5 release.
This fixes: #6787 AND #4164 and adjusts the documentation
Why both? adding custom wheels as requirements requires making those available to the Pyodide environment at runtime. One could place them into the output structure as wheels to be served with the app, but this might clutter it a lot if there are many wheels (and also clash with any bundling system to be implemented). Also semantics with this solution seem clearer: you convert a panel app requiring custom wheels, you get a zip with dependencies and you distribute the app and the zip-file.
As the infrastructure for fixing #4164 was in place with this solution, I then also added a
--resources-flag to the convert-command to include custom resources (such as template files/data files/...) required by the custom-python-portion of the panel app. Everything is packed into a single zipfile, because that's easiest to use with thepyscriptconfig(which only allows to uncompress one zip to a specific wildcard-location); also I bumped the pyscript version, because that feature only got added with one of the small releases.Currently BROKEN: on main, someone updated the version number in
package.jsonwhich is used for getting theCDN_DIST-url. I thinkPY_VERSIONinpanel/io/convert.py(which references a python__version__should also be updated accordingly and possibly also be acquired from a single-source-of-truth!Reasons for restructuring:
collect_python_requirements(finding the requirements or parsing the requirements-file) and some code rewriting/collecting filesystem-URLs.script_to_htmlnow takes a micropip-ready list of package-specifications!Small things:
MIN_VERSIONSfor this reason. Once pyodide implements proper requirements handling this should be put into the requirements-file like for every other project.