Document module dependency matrix

[marcelstoer]

It's helpful I can easily include/exclude particular modules for custom builds using user_modules.h. However, this is always trial-and-error I assume because there's no documented module dependency matrix. If I exclude module X will my NodeMCU binary be broken because module Y depends on X?

This request is to maintain (in wiki) or auto-generate a module dependency matrix.

[marcelstoer]

I was just wondering if that's even necessary. Could it be that all modules are, or have to be, independent of each other anyway?

[vowstar]

TODO:
Build firmware dependent tree and make all modules more dependent.
@MarsTechHAN @vowstar

[MarsTechHAN]

But like SSL, some modules wont use it, and SSL will consume a lot RAM...

How about make a trick?

#ifdef MQTT

#define SSL

#endif

And cross-test will consume some time...We will get this done in a week.

[marcelstoer]

How about make a trick?

MQTT is the only module that fails the build if SSL is disabled, see #432 (there's a note about this in my cloud build). Hence, once this is fixed the SSL setting is totally independent from other settings. IMO that's the way to go.

I'd welcome if you postulate a guideline which says that all non-core modules (e.g. MQTT, 1-wire, I²C, cJSON, etc.) must be standalone and not depend on any other non-core module. Running makedepend or the compiler with the -M option could provide some automatic dependency analysis.

[TerryE]

Ladder logic tricks like this are pretty standard but typically you would have the primary defines as a block and the dependency defines as a second block so in this you'd have something like

#if defined(MQTT) && !defined(SSL)
#  define SSL
#endif

This is the simplest and tidiest way to add such dependences

[confile]

+1

[marcelstoer]

@TerryE @devsaurus @jmattsson @pjsg This starts to bother me more and more and I'm quite convinced that it'll bite us medium-/long-term if we don't tackle this. When I created this issue nearly a year ago we had maybe 20 modules, now there are more than 40.

Initially I thought it would be enough if we had a simple dependency matrix. However, meanwhile we have different sorts of dependencies.

• classical, module A requires module B: currently with Net functionality rewrite on top of LWIP #1379 it looks like you'd need the new tls module for net, http and mqtt if you want secure connections
• exclusive shared resources: hw_timer used by PCM and others
• individual modules intended to be used together: rtctime, rtcmem, SNTP
• de-facto dependencies: RFD - SD Card support #1321, "User is then free to choose how to generate this info. Preferably from rtctime.get()"

Do you think it'd be feasible to maintain a n*n matrix (n=number of modules) and give intersecting cells a specific color (see list above)? If further clarifications are required colorized cells may contain foot note references of course. Can you think of a more sensible approach?

[jmattsson]

Visualising dependencies is tricky. A colour-coded matrix is probably not a bad choice, but I am a bit concerned about its size when we get to e.g. 80 modules. Right now I can't think of anything smarter though, but I'll sleep on it and see if anything pops into my head.

[pjsg]

There are two aspects to this -- one is to make the build work with arbitrary combinations of modules and the other is to make sure all the required functionality is present.

For the first, I think that the code ought to get it right (i.e. have the dependencies in some header file).

For the latter, the situation is a little more complex - but I don't think that there are many cases here. The TLS situation is the most obvious. I don't mind this being explicit.

For the sntp/rtctime case, I'm in favor of just treating these as hard dependencies and using the header file approach.

For the hw_timer case, this is already handled by the build system -- it is always only included if some module that needs it is included (this is the linker magic).

Presumably it would be possible to write a script that, given a list of lua files, extracted all the names that need to be defined and thereby could extract the required module list.

[devsaurus]

A matrix is probably ok for an overview. Though I'm not a fan of encoding information exclusively with colours.
As a con: You'd have to iterate over the matrix again and again in case other modules need to be included due to recommended or must-have dependencies.

If you're looking for a way to guide the user wrt module selection, then the dependencies could also be defined as a set of statements per module. A structured format (JSON?) with dependency type, related module and a description:

• u8g:
  • recommendation spi: some drivers require SPI interface
  • recommendation i2c: some drivers require I2C interface
• pcm:
  • constraint pwm: no parallel operation
  • constraint perf: no parallel operation
• ucg:
  • requirement spi: all drivers require SPI

A local script or nodemcu-build would run through the statements based on the selected modules and provided a summary about

• missing/recommended modules
• constraints for the user application

[marcelstoer]

I am a bit concerned about its size when we get to e.g. 80 modules.

Me too.

For the first, I think that the code ought to get it right (i.e. have the dependencies in some header file).

True but it'd still be nice if we could visualize this for our users. I anticipate questions like "Why is there module X in my firmware, I didn't enable it in user_modules.h?".

For the sntp/rtctime case, I'm in favor of just treating these as hard dependencies and using the header file approach.

AFAIU they're not, it's just that for the anticipated use cases you'll need all of them anyway. Correct Johny?

For the hw_timer case, this is already handled by the build system.

Yes, but I mean to document very clearly that you can't use pcm, perf and pwm at the same time.

...the dependencies could also be defined as a set of statements per module. A structured format (JSON?) with dependency type, related module and a description

I had been thinking about something like that as well. Haven't thought it through all the way, though. It'd most likely be easier to maintain than the matrix and it's got other pros:

• could be injected into the respective module documentation page (similar to the current meta-info table) OR you do it the other way around: main in the docs and optionally generate an aggregated file
• it scales better than the matrix
• you could still generate a matrix from it, whereas the other way 'round would be a little more difficult

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mk-pmb]

Could we maybe borrow the npm module manifest format (package.json) to use deps detection and visualisation tools from the JavaScript community? npm itself is very flexible about package contents, it's not limited to JavaScript modules. The manifest format even officially supports specifying your platforms like

{ "engines" : { "lua" : ">=0.1.0" } }

so no-one will misunderstand the modules as JavaScript modules.

[HHHartmann]

I feel that this is still important. As we do not have individual packages in our C code the npm format seems a bit overdone for this though.
The categories that @devsaurus identified above seem to match our needs.

[czesla]

Hello
I downloaded NodeMCU custom builds by adding 7 modules (file, GPIO, net, node, timer, UART, WIFI) without any additional options.
I launched the ESPFlasher application with the files "nodemcu-master-7-modules-2020-02-22-13-36-35-integer.bin" (434kb), address 0x000000 and "init_data_default.bin" address 0x3FC000.
After starting the board with ESPlorer I get the message:
"Can't autodetect firmware, because proper answer not received (may be unnown firmware)."
I repeated the firmware flashing at the address 0x000000 width "INTERNAL: // NODEMCU" instead of the downloaded "nodemcu-master-7-modules-2020-02-22-13-36-35-integer.bin" and then it was OK, so the ESP8266 system recognizes the firmware.
I tested Firware on two different types of ESP8266 boards (wemos D1 mini and LoLin new NodeMcu V3), each time with the same negative stroke.
I do not understand what the problem is? What i do wrong?

[czesla]

Why the firmware is not working?

[HHHartmann]

@czesla Nice that you are willing to use this project.
I assume that you are using ESPlorer to connect to the esp8266? This program tries to detect some string in the output of the esp. Newer versions of the firmware have changed this output, so ESPlorer detection fails. Try to press the reset button on your esp8266 and you will (most likely) see the output inside ESP8266. If its only garbage try to change the serial speed.

That said, two more points:

• please do not post unrelated stuff to an issue
• altogether this channel is not for support. See https://nodemcu.readthedocs.io/en/latest/support/ for more info

[czesla]

Thank you for your response. I changed the port speed to 115200 and now it's OK. Once again, thank you and sorry for bothering you.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mk-pmb]

I'm still interested.

[nwf]

The correct way to do this, IMHO, is to articulate our build dependencies in the Kconfig language. See #3138 for a stalled effort along these lines.

[mk-pmb]

@marcelstoer would the Kconfig PR be sufficient to make the dependency matrix approach obsolete?

[marcelstoer]

A Kconfig file could serve as the basis to build a dependency matrix. My motivation for this issue 5y ago was to have a visual documentation of the dependencies.

[nwf]

Sorry, to clarify my position: we should have one canonical truth about our intended dependency relationship, and that source of truth should be in Kconfig; any documentation should be generated from this machine-checkable definition rather than being separately articulated. I am, sadly, not aware of a dependency graph generator for Kconfig, but such a thing would be a generically useful tool; there's clearly some functionality along those lines within the Kconfig tool itself because menuconfig can show both forward and reverse dependencies when asked.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mk-pmb]

I am, sadly, not aware of a dependency graph generator for Kconfig,

I'm researching interactive display of directed graphs on websites using JavaScript, so I might build one as an exercise. Where do we currently have the dependency information?