Libraries, Samples and Components: Where next?

[mikee47]

One of the things I like most about Sming is the large selection of libraries and sample applications available out of the box.

However, we cannot just keep adding to these directories indefinitely. I have a few thoughts about where I think we should be heading with this.

** samples **

To reduce the size of this directory, many samples could be moved into an examples sub-directory of the corresponding library. For example, Basic_Capsense, Basic_NeoPixel, Basic_APA102, etc.

This would also make it easier to maintain both library and sample application(s) so they are consistent, and to keep documentation together.

** Libraries **

Every major framework has some kind of library management, so Arduino has its integrated library manager https://github.com/arduino/Arduino/wiki/Library-Manager-FAQ.

This is still lacking in Sming. Rather than just adding new libraries directly into the framework, we should be trying to pare it down to the minimum and use external repositories for all the 'add-on' stuff.

This is one reason why I chose to use an external repository for FlashString, rather than including it directly.
(Note that it's in the Components directory only because it's used elsewhere in the framework, otherwise it would have gone into Libraries.)

With Sming 4.0.0, we now require applications to explicitly identify which libraries it uses (via ARDUINO_LIBRARIES).
This triggers the following actions:

1. Locate the library directory
2. Pull in the library if it is a submodule
3. Pull in any submodules the library uses
4. Build the library
5. Link the library

Step (1) is where further work is required, since Sming knows only about the following Component directories:

• Sming/Components
• Sming/Arch/*/Components
• Sming/Libraries

This list can be expanded by the application COMPONENT_SEARCH_DIRS, but those directories must already exist;
there is no way to tell Sming to look directly in an external repository.

** External Repsitories **

I recently added the RingTone, ToneGenerator and SignalGenerator
libraries, along with some sample applications. Let's consider how these might be moved out of the main Sming repo. and into my own github account, as a separate repository Sming Audio Libraries.

I could just make them submodules, but then that just adds more to the .gitmodules framework dependencies. What I want to do is remove that dependency completely.

As an initial suggestion, rather than listing every individual library we just keep a list within Sming of the location of any 'approved' library repositories. The build system then pulls in the library list from each of these repositories and things proceed as usual.

In this way new libraries could be added to those repositories and would be made available for all users without any change to the core Sming framework. It also provides a way to delegate responsibility and management for those libraries to their originators.

We could also provide a way to allow users to specify additional 'un-approved' repositories for their own uses.

The goal would be to reduce the size of the core Sming repository without affect the user experience.

[slaff]

We need a package manager similar to NPM for NodeJS, Composer for PHP or https://platformio.org/lib. It would be best if we can reuse an existing package manager.

Anyone having experience with package managers that can be used in Sming (C/C++ ) ?

[mikee47]

https://conan.io Looks promising. @aemseemann you mentioned it in the CMake thread, any thoughts?

[aemseemann]

As I already stated in the CMake thread, I have mixed feelings about conan.
It is mainly a package manager similar to what dpkg, rpm, pacman etc. do for Linux.
It will fetch packages (mainly prebuilt libraries/binaries) from one or more remote repositories (sometimes called "artifactories") into a (system-wide) local package cache, from where the packaged libraries, binaries, headers and other artifacts can be accessed.

As such, I can recommend conan for tasks you would normally use your distro's package manager like fetching a prebuild toolchain. However, in the same way as you probably wouldn't include rpm or yum in your incremental development workflow, I would not recommend using conan in that workflow either!

While a 'conanized' Sming (where each library/component is wrapped in a Conan package) might work for projects purely consuming Sming and some libraries/components (which then would all get built in the conan cache automatically), it will be a nightmare if you work on Sming or a component itself, let alone more than one package at the same time. This is because for each modified package, you would have to trigger a package rebuild manually. By default, this is always a full rebuild and even the source is copied from your local editing location to the cache. In addition, to get dependencies right, you have to manually increase the package version with each modification, which effectively creates a new package and also forces a full rebuild.

To be fair, there are ways to overcome the aforementioned limitations like packages in editable mode and, more recently, conan workspaces, but these add further steps to learn and keep in mind all the time. My general feeling is that when including Conan in an incremental development workflow there are a lot of ways to screw up and the benefit is rather limited.

[aemseemann]

I wonder why all the submodules of the top level Sming repositories are submodules. From my understanding, they are not really used in the way git submodules were designed, especially with all the patching done during the build. Wouldn't it be possibly to instead fetch them on demand from an URL/commit HASH given in component.mk without adding them as submodules?
This way one could even extend the Makefiles to support other fetching methods, e.g. downloading a tarball via wget/curl... basically something similar to FetchContent from cmake.

[mikee47]

I wonder why all the submodules of the top level Sming repositories are submodules

Historical!

... fetch them on demand from an URL/commit HASH given in component.mk

I really like this idea! In essence, we'd take the .gitmodules entries and move them into the related component.mk file. I don't think it would be much of a stretch to implement either. It would also simplify things like rebasing.

... extend the Makefiles to support other fetching methods

Why not?!

[mikee47]

Clearly adopting any package manager is going to require customisation, and if it doesn't make our life easier then we don't do it. Something I didn't mention in the opening post was version control, which is kind of important thing to leave out! However, if we ditch (some or all) submodules and go with your hash-based fetching @aemseemann the problem will disappear because the dependencies will no longer be tied into the main Sming repo.

[slaff]

@mikee47 If you want make a PR where the submodules are removed and replaced with a makefile function(s) that take care to fetch the right hash from the right URL. What I would like to remain is a way to fetch all submodules and patch them at once. I need this to create a submodule archive and attach it to every release. This archive is our guarantee that if a remote submodule disappears we still have its last working source code.

[mikee47]

fetch all submodules and patch them at once

Yes, we'll need this to build the docs too.

[mikee47]

@slaff Your submodule archive, do you do that manually? Is that something worth integrating into the build system?

[slaff]

Your submodule archive, do you do that manually? Is that something worth integrating into the build system?

Of course not :) See lines 21 to 29

https://github.com/SmingHub/Sming/blob/develop/.travis/deploy.sh#L21

[mikee47]

Should we be looking at git subtrees instead, or other similar solutions https://codeburst.io/4-git-submodules-alternatives-you-should-know-592095859b0

I haven't used subtrees at all, but the article says "As opposed to submodules, subtrees’ sources files are stored in the repo. It’s not just a link, the code is really there."

I'm not clear ATM how this might affect patching; would it still require patch files?

[aemseemann]

With git subtrees you basically store everything "by value" in the main repository, i.e. it is very similar to having everything in a single repository (as is done for Sming releases).
The only difference is that git "remembers" additional remotes for certain subdirectories of your main repository to make fetching from them/pushing to them easier.
Think of it as an automated way of copying files between the subdirectory of your main repo and a working copy of the linked repository + committing them as a new changeset while reusing (parts of) the original commit message.

With subtrees, patching and the whole "fetching on demand" stuff wouldn't be necessary anymore. However, there is the scalability issue of having everything in one big repository (though git has proven to cope with repositories way larger than what Sming will probably ever grow).

[mikee47]

#1991 is a good case where the sample could be moved into a sub-directory of the ModbusMaster component, so we'd have this:

|_  Sming/
    |_ Libraries/
        |_ ModbusMaster/
            |_ ModbusMaster/   # The submodule
            |_ samples
                |_ simple/
                |_ more-complicated-sample/
            component.mk
            README.rst

[mikee47]

Further to #1991 and, addressing the general case, including sample application(s) in the Component instead of in the Sming samples directory would work just fine, just a few things to note:

1. The sample(s) wouldn't get get built via CI. We need to build the samples ourselves anyway for testing before accepting a PR. However, if something changes in the framework or in a dependent Component which breaks the Library then we wouldn't immediately know about it. It would reduce the load on CI and speed things up though.
2. The entire Component and sample code wouldn't be subject to Sming's coding style.
3. The sample won't appear under samples in the documentation. The Library will still be listed, so the sample documentation would be included there.
4. Each Component can have as many sample applications as it likes, no increased bloat to the main Sming repo.

[mikee47]

#1992 has been an interesting process as we've gone through the creation of a new Component in some detail, thrown up some potential issues and also one interesting and probably simple solution to the 'package management' issue.

Firstly, let's consider that we have an existing Component/Library which we want to move out to an external repository. Here's an outline of the steps required, but so far the only Component like this is Sming/Components/FlashString and I'd hesitate to recommend doing this at present other than an exercise to see how it works.

1. Create a repository for the Component, e.g. github.com/kmihaylov/SmingModbusMaster
2. Add everything from Libraries/ModbusMaster
3. Remove Libraries/ModbusMaster from Sming, including the submodule
4. Add your SmingModbusMaster repo. as a submodule at Libraries/ModbusMaster
5. Put a copy of your Component's configuration file into Libraries/.patches/SmingModbusMaster/component.mk.

This last step is required so that Sming knows about your Component's requirements (submodules and source directories). However, we can add a new make update-components command to do this automatically by pulling in only the component.mk file from all known repositories.

From #1983 (comment) we'd replace .gitmodules with a file listing each Component/Library together with commit hash, etc. The exact structure still to be worked out, but JSON would probably be appropriate and we'd use python to do the work.

[
  "FlashString": {
    "remote":"https://github.com/mikee47/FlashString",
    "commit":"5d128210786c493d45530a4ce23686a9322b8a6e",
    "path":"Sming/Components/FlashString"
  }
]

This would be a lot simpler than submodules as we can see exactly which commit we're using and wouldn't need to involve GIT at all.

[slaff]

The exact structure still to be worked out, but JSON

@mikee47 We can "hack-or-borrow" the existing PHP package manager called composer or at least use some ideas from it as a base. We can use the JSON schema, the central package repository and the algorithm(s) to calculate the dependencies. And the only thing that is left is to write our own cli to replace the original PHP composer that is doing a lot of things that won't be needed for us.

In composer the components describe themselves using a json file. Example: https://github.com/zendframework/zend-db/blob/master/composer.json. There is also a central repository with packages: https://packagist.org/. All actual components that are used end up in a composer.lock file that will contain the information similar to the JSON array from your previous comment.

Composer is created as PHP package manager BUT nothing forbids packages that don't contain a single line of PHP. Also the algorithm that calculates the dependencies is in C (at least that was the case for the Windows client) and we can reuse it as a base. Everyone can publish his/her package - only a valid composer.json file is needed in the root of a github repository.

[kmihaylov]

@slaff, how would Sming's package manager verify if the selected module is compatible with the current version? For example there were a lot of changes to Sming 4 and obviously the modules would not compile without modifications (that would be left to the package maintainer?).

[mikee47]

Worth a read https://github.com/LoopPerfect/buckaroo/wiki/Git-as-a-Package-Registry.

No-one's mentioned Buckaroo yet... https://hackernoon.com/approaches-to-c-dependency-management-or-why-we-built-buckaroo-26049d4646e7

https://github.com/LoopPerfect/buckaroo/wiki/FAQ

[mikee47]

It feels like we're heading in the right direction then.
I've done some work with Drupal in the past - not just for my (empty) website - which uses composer so it's not an unknown.

@slaff Your suggestion of hack/borrow-ing parts of composer is interesting, but what if we could handle all the dependency stuff in a web application? If that were hosted on a public website we could:

• Browse compatible repositories graphically, e.g. 'approved' Sming libraries, other third-party repos.
• Pull in Arduino package information so we can browse that as well
• Incorporate documentation, hyperlinks, etc. for each Component with version information
• Create Component templates
• Integrate with user's github repository to create skeleton repos.

and so on. The generated configuration information could just be cut & pasted into a project.

[mikee47]

re. "Git as a package repository", if I browse to, say, https://github.com/mikee47/FlashString/releases and copy the link for the version 2.0.0 hash, this is what I get:

https://github.com/mikee47/FlashString/commit/98fda8b7c78127758964b79cf4eafb2e47918206

In a nutshell this is all the information needed to incoporate the Component into a Sming project.

Why does it need to be any more complicated than that?

[mikee47]

I was wondering why buckaroo was so-named, it's because it uses Facebook's Buck build system. That brings us back to CMake #1684, since my initial thought is that we wouldn't want yet another build system for Sming. So, then, might Buckaroo with Buck fullfil both requirements? Or are they different?

[aemseemann]

In PR2006 (Comment), the question was brought up why there is a distinction between Components and Libraries. Quoting from that comment, the main issue is confusing variable/file names:

• We have to use ARDUINO_LIBRARIES also for libraries that have nothing to do with the Arduino ecosystem
• Both libraries and Components use a component.mk file.

The distinction between "Core" parts and optional parts could be resolved by an appropriate directory hierarchy instead. ESP-IDF also knows about 'components' only.

Sticking to the existing terms for the moment, another step probably worth considering is pulling more optional features out of the core into Libraries. Some candidates that come to my mind:

• Services/CommandProcessing
• MQTT features
• More network protocols like FTP, SMTP, maybe even HTTP

[slaff]

In IMHO the way forward is

• Have components only. Deprecate ARDUINO_LIBRARIES and add backwards compatibility. And phase out ARDUINO_LIBRARIES somewhere around Sming version 4.5.0 or 5.0.0.
• Providing "slim" core and moving some features to components is a step forward. The next step would be to be able to download only the slim core and fetch the optional core or non-core components on demand.

[mikee47]

From #1431:

Libraries

Code quality is hugely variable. There should be two library folders, one for high quality code and another for everything which probably needs some attention. Makefiles should be updated so user code isn't broken but if I wish to build Sming with only the good libraries then it makes that much easier.

ArduinoJson, for example, is a high quality module which would be classed as a 'core library'. Attributes for a HQ module would include:

• Proven track record
• Written to a good standard (even if it's not the same as the Sming standard)
• No timer-driven polling; code should execute asynchronously using callbacks to perform specific tasks, except where alternatives are not available. For example, polling for keypresses via SPI or I2C might be necessary if there is no interrupt line available.
• No delay loops or calls to watchdog timer: both of these violate the core principle of Sming.
• Ideally, written to be cross-platform so it can be tested under linux/mingw build.

It would be preferable to treat compiler warnings as errors, however many of the libraries would fail to build.

@todo Perhaps 'unapproved' libraries should be built into a separate library, against libsming.

Fast forward 18 months and we have made some progress. If you run make list-components the first entry looks like this:

COMPONENT_SEARCH_DIRS
- /home/mike/sandboxes/sming-dev/Sming/Arch/Host/Components
- /home/mike/sandboxes/sming-dev/Sming/Components
- /home/mike/sandboxes/sming-dev/Sming/Libraries

This is in priority order, so core components in the first two locations will always be picked up ahead of anything else.

The point I'd like to make here is that whilst all of these may be Component objects (if you will), they are in different categories, and that is the important distinguishing factor.

If we just used COMPONENT_DEPENDS, we could qualify the Component names to convey this information, e.g. core/ssl, library/IR, library/beta/Mirf, etc. The category could just map onto a subdirectory path.

[aemseemann]

If we just used COMPONENT_DEPENDS, we could qualify the Component names to convey this information, e.g. core/ssl, library/IR, library/beta/Mirf, etc. The category could just map onto a subdirectory path.

👍
This would also allow architecture-specific libraries.
Maybe some more categories could be introduced, e.g. net/{mqtt,ftp,...}, hw/{Adafruit_Neopixel, ...}, gui/ITEADLIB_Arduino_Nextion, ...