Make cmsis-pack-manager optional

[flit]

This patch allows pyOCD to handle cmsis-pack-manager not being installed. This is currently necessary to use pyOCD on Raspberry Pi, since CPM fails to build on that system. CPM is still listed as a requirement, so docs were added for how to install and use pyOCD in this way.

[flit]

/morph test

[cederom]

Hey @flit the latest CPM release allows now compilation on all Open-Source platforms even with pip.. some dependencies needs to be installed though (Rust) :-)

[flit]

Ohh?? I totally missed that. But I thought that Rust still ran out of memory when building CPM on RasPi (for example), as described in pyocd/cmsis-pack-manager#108.

[carlosperate]

If the source is on PyPI and the wheel can be built using pip wheel cmsis-pack-manager then PiWheels would attempt to build it automatically: https://www.piwheels.org/project/cmsis-pack-manager/
Don't know if it current supports packages with rust source code, but worth checking out.

[flit]

@carlosperate Thanks, I hadn't heard of PiWheels, that's really useful.

[flit]

CPM needs to be released as a source distribution before piwheels would work. Assuming they support building with Rust, which I haven't checked yet. And assuming the build would pass.

[cederom]

Its already here @flit, I have tested that on a FreeBSD in May, which did not work before, @theotherjimmy updated the CPM that fast :-) pyocd/cmsis-pack-manager#97

https://github.com/ARMmbed/cmsis-pack-manager/releases

https://pypi.org/project/cmsis-pack-manager/

[flit]

I don't see a source release for CPM for the 0.2.8 release: https://pypi.org/project/cmsis-pack-manager/0.2.8/#files

In addition, PiWheels shows "binary only" for all of the releases.

[cederom]

Uhm.. true.. no sources on the PyPI yet.. sorry @flit.. I have compiled and tested it by hand on FreeBSD from sources before the release just to make sure it works..

Probably @theotherjimmy did not upload the source package yet? Any chance to upload the 0.2.8 sources of the CMSIS PACK MANAGER to PyPI please? Do you need any help with that @theotherjimmy? :-)

Anyways this optional CPM feature seems nice to have when CPM is not there for any reason.. I have experienced that before that I could not use pip install pyOCD because there was no CPM dependency on my host (#674).. Would that have to be solved on setup.py / pip install level anyhow..?

[theotherjimmy]

I was under the impression that a source-only release was not required for that issue. I can make one if it's needed.

[cederom]

Not exactly "source-only".. adding source code package to the PyPI in addition to a binary builds will allow people to build CPM on their own and on platforms where WHL binary is not provided (i.e. RaspberyPi, BSD, etc).

Still local Rust install must be present on a target machine for pip install cmsis-pack-manager to build from the sources!! But we have source code that builds with stable rust so it should work everywhere now :-)

We have already tested that build on FreeBSD but that was manual work on files unpacked from GitHub Release Repo. Source code package on PyPI is still necessary for pip install to work.. That would be really nice to have please @theotherjimmy - seems like only like putting the tarball to the pypi..? :-)

[theotherjimmy]

Yeah, I'll put the tarball from 0.2.8 on PyPi today.

[flit]

@theotherjimmy Thanks!

[theotherjimmy]

Alright, I uploaded 0.2.7 and 0.2.8 as an sdist. I also opened pyocd/cmsis-pack-manager#116 to enable sdist in the future (it turns out it's not that hard)

[flit]

Thanks for uploading. Unfortunately, but as expected, piwheels failed to build it. But had to have the source first!

[theotherjimmy]

@flit How can I make that work as expected? Could it use the uploaded binary instead of trying to build itself?

[flit]

Unfortunately, I don't think there's anything you can do… I created piwheels/packages#36 to report the build failure. They may not even support Rust yet, dunno. You can't upload your own binaries to piwheels. And, the piwheels FAQ points out that you really shouldn't upload RasPi wheels to PyPI due to Arm architecture weirdness they use to achieve compatibility across RasPi versions.

The best we might be able to do for the time being is build a CPM RasPi wheel and host it on GitHub or somewhere. Then users would have to manually install CPM via pip from this wheel prior to the usual pyOCD install.

[cederom]

Hey @flit @theotherjimmy for me building CPM on FreeBSD from sources works with a simple pip install pyocd: https://asciinema.org/a/nrZwa86mxJBmcp5EyPyHKop5p

We just need to update package version in setup.py on each release or use some external file with a script that would provide version number based on a git tag when creating source package :-) At the moment pip thinks it is dealing with 0.1.1 instead 0.2.8 but it works as expected :-)
Update: I can see you are working on this right now at pyocd/cmsis-pack-manager#117 :-)

What exact problem do you encounter? I will try to build it on a BananaPI in a moment just to make sure and report back :-)

[cederom]

Oookay.. long time no see Linux.. all seems different here ;-) I also encounter problems to build CPM on Linux Armbian / Banana Pi M1: https://asciinema.org/a/VpeUUMqtbPAXCYrfpnRg27cMj

I had to update my date by hand with additionally installed ntpdate utility because working in the background ntpd did not set time correctly what caused basic SSL issues wow ;-)

[cederom]

I have managed to build CPM and pyOCD on Armbian Linux / Banana Pi M1 using pip install pyocd :-)

On Linux some additional development dependencies are required in order to build CPM: rustc, cargo, python3-venv, python3-dev, libffi-dev.. maybe something more :-)

https://asciinema.org/a/MeYMuwJkUxAFhMG5WwVjJ5aZP

[saper]

I ran into this today on FreeBSD (no rust installed yet) and I was wondering is there any issue in making CPM an optional dependency and make it required only when installing, say pip install pyocd[cpm] ?

[flit]

Hi @saper, I'm afraid that would be problematic. CPM is required for the managed CMSIS-Pack functionality (pyocd pack), which is how many users install target device support. And the vast majority of users don't have issues with installing CPM (afaict).

I'd prefer to somehow make CPM an optional but always attempted dependency, i.e. where a failed install is allowed. So far I haven't seen a way to do this with setuptools, but honestly I haven't looked very deeply either.

I'd also quite like to ship wheels for FreeBSD, but there doesn't seem to be a standard.

Anyway, rust support is becoming required very quickly on all platforms.

[cederom]

Hey @saper I am working on a FreeBSD too and I can build pyOCD with some dependencies provided by the system - all are already there in Ports tree [1] :-) There is a separate dedicated thread for this on CPM project page [2]. The solution would be to provide binary compiled CPM on a pip repo for FreeBSD so we could just pip install pyocd even without compiling it..? If so the question is to @flit can we create/attach additional build machines for FreeBSD CI?

[1] https://asciinema.org/a/nrZwa86mxJBmcp5EyPyHKop5p
[2] pyocd/cmsis-pack-manager#97