Communication protocol simplification possible?

[ralphwetzel]

@phoddie --
Currently the status message consists of the status data (necessary for node-red) and an additional object that provides source information. The (only?) relevant property of the source object is the id of the emitting node, that is used by the plugin to build a status notification that is understood by the editor.

node-red-mcu/nodered.js

Lines 296 to 306 in 7ddfd04

	status(status) {
	const msg = {
	status: {
	...status,
	source: {
	id: this.id,
	type: this.constructor.type,
	name: this.name
	}
	}
	};

Thus type and name are not used & can (as well) be derived from the provided id by the node-red runtime. They yet need to be processed & transmitted with each status message - adding a significant load without value.

I propose to change the protocol to just provide the node's id & omit either type and name. I'm fine with keeping the property named as source, whereas id would fit as well:

 status(status) { 
 	const msg = { 
 		status: { 
 			...status, 
 			source: this.id
 		} 
 	}; 

This proposal is of course valid not only for status, but for all messages traced back to the node-red runtime.

[phoddie]

To start, I think we should review the message and how it is used. The message is a match for the message that Node-RED sends to Status Nodes when the status is changed. Here's a flow to explore that.

[
    {
        "id": "2fec9e130ad0d45f",
        "type": "tab",
        "label": "delete me",
        "disabled": false,
        "info": "",
        "env": []
    },
    {
        "id": "5e3247acce2fd674",
        "type": "status",
        "z": "2fec9e130ad0d45f",
        "name": "",
        "scope": null,
        "_mcu": false,
        "x": 480,
        "y": 220,
        "wires": [
            [
                "16b7efbf68b385c1"
            ]
        ]
    },
    {
        "id": "16b7efbf68b385c1",
        "type": "debug",
        "z": "2fec9e130ad0d45f",
        "name": "debug 8",
        "active": true,
        "tosidebar": false,
        "console": true,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "_mcu": false,
        "x": 670,
        "y": 220,
        "wires": []
    },
    {
        "id": "1790bba3c0262e5c",
        "type": "inject",
        "z": "2fec9e130ad0d45f",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": true,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "_mcu": false,
        "x": 170,
        "y": 360,
        "wires": [
            [
                "9afeee4e3f8febbb"
            ]
        ]
    },
    {
        "id": "9afeee4e3f8febbb",
        "type": "function",
        "z": "2fec9e130ad0d45f",
        "name": "function 3",
        "func": "node.status({fill:\"red\",shape:\"ring\",text:\"disconnected\"});",
        "outputs": 0,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "_mcu": false,
        "x": 340,
        "y": 360,
        "wires": []
    }
]

In Node-RED, it outputs the following to the console:

29 Oct 17:45:37 - [info] [debug:debug 8] 
{
  status: {
    fill: 'red',
    shape: 'ring',
    text: 'disconnected',
    source: { id: '9afeee4e3f8febbb', type: 'function', name: 'function 3' }
  },
  _msgid: '6c17a86e265510f2'
}

Using Node-RED MCU it outputs the following to xsbug (I've reformatted it for readability).

{
	"status": {
		"fill": "red",
		"shape": "ring",
		"text": "disconnected",
		"source": {
			"id": "9afeee4e3f8febbb",
			"type": "function",
			"name": "function 3"
		}
	},
	"_msgid": "287b8cad8de8ef90"
}

This same message object is also sent to xsbug to relay to the Node-RED Editor. Changing the message propagated to Status Nodes will break compatibility with Node-RED.

The message sent to xsbug could be simplified by removing unnecessary fields - in addition to the source type and name, it seems possible that the _msg ID is also unnecessary. But that would need to be done in such a way that the messages sent to Status Nodes maintain compatiblity.

As an aside, you mention "adding a significant load without value". I'm curious if you are able to quantify the load impact?

[ralphwetzel]

Speaking w/ data: The total of all proposed simplification shortens the message shown above from 417 chars to 253 chars; 40% less data to process & transmit.

<xsbug>
<bubble name="43232f662ac1fdac" value="1" path="/Users/ralph/Projekte/node-red-mcu-plugin/node-red-mcu/nodered.js" line="289">{&#34;status&#34;:{&#34;fill&#34;:&#34;blue&#34;,&#34;shape&#34;:&#34;dot&#34;,&#34;text&#34;:1}}
</bubble>
</xsbug>

[phoddie]

It is an interesting idea to use the optional conversation argument of trace.left to carry the node ID. That allows filtering in the xsbug message pane. It could be OK, but first I have a few questions / observations.

1. Is it meaningfully smaller? I think it does save a few bytes.
2. It makes the message sent by Node-RED MCU Edition more unlike the message in Node-RED. Generally I've tried to keep them the same to minimize the chance to translation errors. That's for you to manage. ;)
3. If this change is made, corresponding change to the Debug Node's onMessage method should also be made so the protocol is consistent. Correct?
4. The biggest win remaining to reduce the payload size is be to suppress the path and line attributes, but that would require deeper changes and potentially modifying xsbug too.

[ralphwetzel]

2. That's for you to manage. ;)

👍 A bit more effort for the plugin - but no big deal, and already done.

3. If this change is made, corresponding change to the Debug Node's onMessage method should also be made so the protocol is consistent. Correct?

Yes. That makes sense.

4. The biggest win remaining to reduce the payload size is be to suppress the path and line attributes, but that would require deeper changes and potentially modifying xsbug too.

This is going to lead to a dedicated protocol for the Node-RED communication. Whereas I confirm the additional potential, I have the impression, it's - currently - not demanded. I'll do some further stress-testing the next days; perhaps this gives us any supportive data...

[phoddie]

I've committed a small change to set the conversation to the node ID on the messages for the editor (debug and status). Because this is a breaking change to the protocol, I did not remove the id property from the source yet. Once you have updated the plug-in to depend only on the conversation value as the ID, let me know and I'll remove those. I also have some work planned to improve the debug node output that I'll do at that point.

This is going to lead to a dedicated protocol for the Node-RED communication

FWIW – I don't think it needs to go that far. The xsbug protocol has proven to be remarkably robust and flexible. That's good, because it is also deeply intertwined with XS itself, so it isn't practical to make significant changes.

I would like to propose one additional change that will require support from the plug-in. When flows are running and the editor is not attached, the MCU is still preparing and sending messages to the editor. That's unnecessary. A lightbulb running Node-RED won't always be connected to the editor. ;) So... I'd like to propose that the plug-in sends a message to Node-RED on the MCU when it connects. That will be a signal to the Node-RED MCU runtime that it should send messages for the editor. That message can be delivered using the same mechanism as "inject". The payload can simply be an empty object (which will be ignored now because it has no flow or Id). What do you think about adding that to the plugin?

[ralphwetzel]

Once you have updated the plug-in to depend only on the conversation value as the ID, let me know and I'll remove those.

Done now. The plugin understands both protocols.

the plug-in sends a message to Node-RED on the MCU when it connects

Of course. I would even go a bit further: Let's change the default status from "MCU sends data." (as it does today) to "MCU doesn't send data." Then it's up to an external trigger to motivate the MCU to change into sending state. One message of the plugin could e.g. be worth 60 seconds of data sent by the MCU. Once this time is over, the MCU has either received another external trigger (restarting the 60s period) - or falls back into non-sending mode.

[phoddie]

Done now. The plugin understands both protocols.

Cool. Thanks. I will try to updated the MCU runtime for that this evening.

Let's change the default status from "MCU sends data." (as it does today) to "MCU doesn't send data." Then it's up to an external trigger to motivate the MCU to change into sending state.

Exactly. The empty message from your plug-in is a signal to send the state to the editor.

One message of the plugin could e.g. be worth 60 seconds of data sent by the MCU. Once this time is over, the MCU has either received another external trigger (restarting the 60s period) - or falls back into non-sending mode

This is additional complexity and a bit of memory for a relatively obscure situation. Let's hold onto this idea. If there's a clear need, it can be implemented then.

[ralphwetzel]

Ok & sorry! The misunderstanding is on my side. Now I got it; my intension was a completely different one.
Let's see, what's necessary to implement this feature...

[ralphwetzel]

Let's see, what's necessary to implement this feature...

Two topics I'd like to mention already:

• There's a data property in options that was foreseen to transmit command parameters.

node-red-mcu/nodered.js

Lines 489 to 492 in 93c9f67

	onCommand(options) {
	if ("debug" === options.command)
	this.#active = !!options.active;
	}

Line 491: this.#active = !!options.data;

• I became a bit confused as the debug messages continued to appear in the xsbug Console - despite being switched off in the editor. Wouldn't it make sense to consistently disable these messages - neither display them in xsbug, nor send them back to NR?

[ralphwetzel]

The functionality should be available with the latest commit. This yet requires the small change in line 491, as shown.

[phoddie]

Nice. I've pushed the change from active to data.

I became a bit confused as the debug messages continued to appear in the xsbug Console - despite being switched off in the editor. Wouldn't it make sense to consistently disable these messages - neither display them in xsbug, nor send them back to NR?

I could go either way on the behavior. Since continuing to log in xsbug confused you, let's try suppressing those for now to match the Editor.

[phoddie]

BTW – it is pretty cool that the buttons of both the Inject and Debug nodes in the Editor are handled by the MCU. Nice work.