Reliable PC-Hub-communication protocol - options to handle it in C?

[Novakasa]

Hi all,
Since we now have micropython module support (awesome!), I have spent a few days extracting the PC-Hub-communication protocol of my hub programs into a separate module. While I was at it I wanted to upgrade it's reliability and efficiency, enabling messages of pure binary streams, using a xor-checksum as well as fail-safe timeouts if the messages somehow get desynced with the receiver buffer.

micropython communication modulle
Example micropython program using the module
Python program with the PC-side protocol and a program to test the above program

This works pretty well, however the resulting module is now quite large ( ~2-3kB on the hub). This seems to produce problems for my more complicated programs, which result in a final size of ~6kB and the cityhub crashes the program in hard to debug ways. I am guessing it is a memory thing, since I don't have the issue on the TechnicHub.

My question here, more than actually understanding the memory-issue, would be what my options are regarding moving this protocol-layer to C code. Is the firmware modular in some way where I could just compile the firmware with every new release and just adding the protocol module? Another option would be a fork of the firmware of course, or, if there is interest, I could try to make it an actual PR to the firmware. What are your thoughts?

Edit: Of course all of this assumes that implementing the module in C would make it noticably more efficient. I am guessing so, but admittedly I am still thinking about CPython-level efficiency while working on Micropython programs, which might not be entirely accurate.

[laurensvalk]

Thanks for testing advanced code and sharing your findings with us!

the resulting module is now quite large ( ~2-3kB on the hub). This seems to produce problems for my more complicated programs, which result in a final size of ~6kB and the cityhub crashes the program in hard to debug ways.

Do you get any kind of exception or does the hub just switch off?

Python program with the PC-side protocol and a program to test the above program

This seems to link to the same program as the first one listed.

My question here, more than actually understanding the memory-issue, would be what my options are regarding moving this protocol-layer to C code.

There is a way to "freeze" MicroPython code along with the firmware. That way you don't have to write it in C but you still get some of the benefits. I don't think is set up right now but we should be able to do that. If set up, you can basically put your Python libraries in a folder and it gets compiled and built into the firmware as a frozen module. Would this be useful?

See also optimizing. I haven't tried any of these things recently, so I'm not sure what is or isn't enabled right now.

If you find a useful feature that isn't enabled or configured, it should be possible to enable it, so feel free to open an issue.

[laurensvalk]

Thanks, I was mainly curious about whether Pybricksdev was used or a slightly simpler (if less automated) approach like this but it sounds like you're already handling things byte by byte either way.

Still, this approach can be useful in case you want to rule out Pybricksdev side effects.

since I hijack all output from the hub

It would indeed be nice if the NUS service is completely separate from standard IO so you can use both without interference. We might look at that in a future release.

[Novakasa]

There is a way to "freeze" MicroPython code along with the firmware. That way you don't have to write it in C but you still get some of the benefits. I don't think is set up right now but we should be able to do that. If set up, you can basically put your Python libraries in a folder and it gets compiled and built into the firmware as a frozen module. Would this be useful?

This sounds really useful to me, since I would be using this module on all my hubs.

It would indeed be nice if the NUS service is completely separate from standard IO so you can use both without interference. We might look at that in a future release.

Yeah, it was a bit of a headache to work around the standard io, but I got it in the end. I might be able to use a bit more flexible protocol if we separated them.

[Novakasa]

If we were to freeze a large module, according to the linked docs it'd be stored with the firmware. Would that actually benefit the RAM usage? I am guessing the situation has changed since #483 (reply in thread), as the linked issues therein have been closed.
Edit: Seems that way, according to this micropython commit message

[laurensvalk]

See these two paragraphs:

We only do the first step. The .mpy is still loaded from RAM. But the second step will have additional savings, since the script will no longer be in RAM:

[laurensvalk]

We used to have a folder called "frozen" in the each of the bricks/ folders, and something in the Makefile that would freeze all of the contents into the firmware. We can probably restore that. It shouldn't take any space for anyone who isn't using that feature.