Add make target python-requirements to install Python requirements of project

[aemseemann]

This is a proposal to add a new make target to installs all python requirements of the project from 'requirements.txt' files of the Components it depends on.

Open tasks I would like to collect some feedback first:

• CI server integration: Since a generic target that invokes python-requirements for every sample is quite slow, it's probably best to explicitly invoke it only for those samples known to have non-standard python requirements.
• Check if more Components/Samples could make use of this mechanism. - Done: Full-text search for 'python' revealed just the terminal Component and the HttpServer_FirmwareUpload sample.

[slaff]

@aemseemann Let's split this PR. Move all stuff related to the configuration of the python interpreter into a separate PR (for example: feature/config-python-interpreter). And then rebase this PR on top of the feature/config-python-interpreter PR.

This is a proposal to add a new make target to installs all python requirements of the project from 'requirements.txt' files of the Components it depends on.

What will happen if a component has example that requires python modules, but Sming and the current project itself do not require it? Do we have to explicitly list all requirements.txt files or what would you suggest?

[mikee47]

Instead of explicitly invoking make python-requirements, we could arrange things so this is done automatically on first use. With submodules, we do this by touching a .submodule file so perhaps a similar scheme could work for this?

[aemseemann]

@slaff: for the PYTHON config PR see #2022 .

What will happen if a component has example that requires python modules, but Sming and the current project itself do not require it? Do we have to explicitly list all requirements.txt files or what would you suggest?

(Hope I understood your question correctly) The make target is supposed to install the python requirements of the Sming project it is invoked on. The requirements.txt of all Components/libraries directly or indirectly consumed by the project are collected, either by autodetecting a requirements.txt in the Component's root dir or using the file(s) explicitly provided by the Component via COMPONENT_PYTHON_REQUIREMENTS. The latter allows the requirements to depend on configuration variables. See #2006 for example, where the OtaUpgrade library requires pynacl only if signing and/or encryption are enabled.

If a library/Component comes with an example project ('example' subdir in tree below), its Python requirements are irrelevant for the project consuming the library/Component and hence are not installed.
However, if the example is itself a Sming project it can provide its own requirements.txt (in addition to the requirements file of its parent library).

Sming/Libraries/MyLib
|- example
| |- ... (sources, component.mk, Makefile including $(SMING_HOME)/project.mk)
| |_ requirements.txt
| - ... (component source)
| - component.mk
|_ requirements.txt

[aemseemann]

@mikee47:
I don't think it is a good idea if the build system does something unexpected, at least not automatically, and modifying the user's system (everything beyond the working copy) would certainly qualify as unexpected IMO.

There are too many ways the user might want to manage its python installation. Some users might be fine with a user/system wide installation. Some might prefer a virtualenv (for this project only or shared with other Sming working copies...). Others might want to use Distro packages instead of pip.

For the latter kind, it might be helpful to add another target to just dump all the requirements.txt files relevant for the project. <-- Edit: done with commit bb48e01: make list-python-requirements

[aemseemann]

* CI server integration: One idea to add a similar target that recurses into every sample and invokes `python-requirements` for each of them.

I tried to add a python-requirements target to $(SMING_HOME)/Makefile that recurses into every sample (similar to make samples). However, that target takes quite some time to complete, even if all requirements are already installed, so I think for the CI build scripts, it is probably best to stick with the current solution and invoke make python-requirements only where needed (currently only HttpServer_FirmwareUpload).

[mikee47]

Doesn't work with MinGW. Why not just COMPONENT_PYTHON_REQUIREMENTS := ...

[mikee47]

(Specifically, undefine not supported in GNU Make 3.81)

[aemseemann]

The difference between an undefined vs. an empty COMPONENT_PYTHON_REQUIREMENTS is auto detection of 'requirements.txt', i.e. in the empty case no auto-detection is performed.

In the latest commit, I moved auto-detection before component evaluation. This way, the component can simply clear the variable if it wishes to suppress an auto-detected requirements.txt. No undefine necessary & successfully tested with make 3.81.

[mikee47]

... and ifndef COMPONENT_PYTHON_REQUIREMENTS ?

[aemseemann]

That would have much clearer, but no longer necessary.

[aemseemann]

@mikee47: The Travis build failure seems to be related to the install step of the python requirements for building the docs. Unfortunately I do not understand what the actual problem is. Can you shed some light on what's going on there? (If I remember correctly, it's not the first time I encounter this error, but it always disappeared when triggering another build.)

[mikee47]

@aemseemann Checking back through the logs the error: invalid command 'bdist_wheel' has been present for months, but doesn't usually cause a build problem (even with the docs). I've got it fixed by installing extra packages, etc. now just whittling it down to find the ones we actually need.

[mikee47]

@aemseemann I appear to be able to push commits directly to your branch... that was initially by accident but hope it's OK to do it here?

[aemseemann]

@mikee47: No problem for pushing directly to the branch, but it is quite strange indeed. I cannot see any setting that would allow this. EDIT: It's not that strange after all: By ticking "Allow edits from maintainers" for this PR I've granted push permissions to anyone with write access to the upstream repository.
Anyway, thanks for fixing the build. Now it's up to @slaff to decide if the doc fix can remain in here or should be factored out into another PR.

[slaff]

@aemseemann @mikee47 What is the state of this PR?

[aemseemann]

@slaff: If you agree that @mikee47's fixes to the random documentation build failures can stay part of this PR, then its ready from my side.