This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
Porting to C #2
Comments
|
Yes, it can, and it will, given that I have the resource to do it. It has always been my intention to create a version in C. When I have added the last few features to the Python reference implementation, and I lock in the API, I will start work on the C implementation. |
|
Also, I would very much appreciate help on this endeavour, so if you want to join in, you are very welcome. I do however think it’s necessary to wait until the Python implementation is done, the API has been locked in, and the documentation has been completed. |
|
Okay ping me once you're ready to create the C version. I've listed some interesting ideas on the Meshtastic forum. |
|
Are you still maintaining it? |
hard to tell , no updating since last August. Let's find out is there anything we can do? |
|
Well first of all check if the current version is reliable enough to be used, then start porting it to C using a proper HAL in order to support as many embedded platforms as possible. |
sounds like a plan |
|
I am still very much maintaining Reticulum. Life has not been too generous with time lately though, and I've had to focus on things more related to survival than passions. Although I had a goal of having Reticulum in an API-stable beta release by the first day of 2021, it did not pan out that way, and I was severely delayed. Things have calmed down for me now though, and for the next few months, I should be able to complete that work. I'd really strongly suggest against starting a C-port right now. There's still a few crucial details that need locking in, and doing this without the API documentation I'm currently writing would be a frustrating experience. I would very much appreciate the help once it is time, but please have just a little more patience :) I know it's been a long wait, but it will be worth it very soon. |
|
And in the meantime I am open for any discussion on the matter in this thread. I will be available to answer questions and discuss it here. |
How nice, happy to know you were back. It's ok, we will wait for your docs and lock in the few crucial details |
|
Hello everyone who is still listening here. Today, I released Reticulum 0.2.0. This is the first release marked as being in beta state. This also means the API documentation and manual has been published at: https://markqvist.github.io/Reticulum/manual/. In a sense, one could at this point start writing a C port. In the service of being as transparent as possible, I will have to say that I am planning a very big change to the internals of Reticulum though, which will greatly impact any implementations. I am 95% sure that I will change Reticulum to completely use Elliptic Curve Cryptography on Curve25519 for all assymetric cryptography operations. This will mean RSA is no longer needed, which has been a very limiting factor in the project so far. By using Ed25519 signatures and X25519 ECDH key derivation instead of RSA, protocol overhead will be greatly reduced, and security will be staggeringly enhanced. It really seems like a win-win, but when I worked on the theory behind Reticulum 5 years ago, I didn't know enough about elliptic curves to utilize them confidently, and as such chose to go with RSA-1024 instead. I'm currently researching the last 5% of doubt I have about this decision, and one of the topics I'm looking into is the feasibility of running X25519/Ed25519 on low-power MCUs and what C implementations exist. I am very interested to hear anyones thoughts on this. |
|
Hey, first of all nice to have you back and to see Reticulum is under development. Going to the actual task of porting it to C, we need to define some common guidelines beforehand, such as using size-constrained integers ( |
|
The m parameter is not appended into the message, because it is already there in the header field ;) Byte two of the header field is the "hops" byte, which tells how many hops a packet has taken, which is also true for announces. I updated the "wire format" section in the manual to make it more clear. Ok, I just finished moving everything to elliptic curves. Was actually a lot less work than I had anticipated. And as you say, it seems there is good support on microcontrollers and in C, so this is going to be the way forward. Protocol overhead has also been greatly reduced now. It's all very, very nice. What you're saying regarding locking in the typing definitely makes sense, as does having a way to control dynamic vector allocation. I'm going to take a break for a couple of days now, cause I'm literally exhausted ;) When I start again, I will start mapping out the outline of the C version. I'm quite hesitant to use a RTOS, since I've been down that road before a long time ago, and in the end it ended up being much more hassle than help. My current stance is that making the C port as dependency free as possible is the way to go. We'll need crypto libraries, yes, but that should be more or less it. Task scheduling and async io should be possible with an internal job handler, since the internal "transport machine" of Reticulum is actually pretty simple. You're welcome to convince me otherwise though :) |
|
Happy to hear that. I am here still listening |
|
Ok moving forward, I've created a pull request to act as WIP for us all. Other general things we need to think about before starting are:
Small OT: have you seen the Helium project (www.helium.com)? |
|
My suggestion is porting to c++, instead of porting to c. Using libs like Qt or boost. To reduce pains and acquire similar performance at the same time. Actually I have already done some porting work with Qt. It's really time saving, and you have QString, QCryptohash, logging system, almost everything you need.
在 2021-05-27 20:30:28，"Giammarco R. Casanova" ***@***.***> 写道：
Ok moving forward, I've created a pull request to act as WIP for us all. Other general things we need to think about before starting are:
inline logging: Serial print, file log, other?
classes: do we keep them or use a purely functional C-style?
string and number manipulation: any good replacement for native Python libraries?
Small OT: have you seen the Helium project (www.helium.com)?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
Sorry but that’s a huge red flag as neither boost nor qt work in embedded microcontrollers like the ESP32 |
I was thinking the code will run in host boards like raspberrypi or a pc。Are we going to put the mac layer and application layer all in MCU? |
And by the way, maybe it worth giving a look at Qt for MCU. That it says can run in small systems with 80KB MEM minimum and might include some libs I mentioned above, and even with GUI. See https://www.renesas.com/us/en/document/prb/qt-mcus |
|
Porting to C/C++ makes no sense if you run on a single board computer like Raspberry Pi, Beaglebone or similar as they can perfectly run the Python code too. My idea behind moving to C was to support uC like the ESP32, a de-facto standard in IOT and possibly some boards from the Arduino family. Luckily, Arduino libraries already come with most of the things we need and natively support the ESP32. |
|
Alrighty, let's get a few things straight. I've been a bit out of the loop here, since I'm taking a break with family, so this will be a bit short. But there's a few key points I want to make: As @grcasanova says, there's no point in writing a C port targeted for Python-capable boards. The whole purpose of a Reticulum implementation in C is to support micro controllers and systems running on bare metal. This means that the implementation should be as dependency-free, portable and general purpose as entirely possible. Qt is great and all, but it has absolutely no place here. I'm not gonna try and stop anyone who wants to write a C++ implementation of Reticulum using whatever frameworks they desire, say Qt, but it should not be part of this repo, and will be a concurrent project. I'd welcome such a project, this is just not the place for it. In my view, the goal of the C implementation is a portable library that can be utilised on µCs, bare-metal systems and similar, or be included in compiled C/C++ programs. From my own perspective, the reason for doing this is to allow small, power-efficient and portable devices to utilise Reticulum for connectivity, and that is my motivation for pursuing this goal. Such devices can include handheld communication devices, but also small integrated Reticulum Transport nodes (imagine a small solar-powered Reticulum node with a LoRa transceiver, that can cover many square kilometers with connectivity). For things as this, we need the C library. Currently, the only areas where I can see the need for external libraries are the cryptographic primitives needed to assemble Reticulum. I think we can make do with standard C library for the rest. Maybe, maybe not. We'll see, and that's something that needs to be determined and settled on before any code is written. Other points that need careful thought are:
In my opinion, we need clarity on all of these points before any code is written. |
|
I totally agree for targeting bare metal and possibly avoiding libraries that may not be supported.
|
|
Agreed. So now we are clear that we are porting to C or C++ for MCU like ESP 32 board. And the first question in front of us is which coding language we choose. I vote for C++. No worry about performance compared to C. The only concern is the Size of the Firmware. Will it be too large, When choosing C++.
在 2021-05-28 19:15:52，"Mark Qvist" ***@***.***> 写道：
Alrighty, let's get a few things straight. I've been a bit out of the loop here, since I'm taking a break with family, so this will be a bit short. But there's a few key points I want to make:
As @grcasanova says, there's no point in writing a C for Python-capable boards. The whole purpose of a Reticulum implementation in C is to support micro controllers and systems running on bare metal.
This means that the implementation should be as dependency-free, portable and general purpose as entirely possible. Qt is great and all, but it has absolutely no place here. I'm not gonna try and stop anyone who wants to write a C++ implementation of Reticulum using whatever frameworks they desire, say Qt, but it should not be part of this repo, and will be a concurrent project. I'd welcome such a project, this is just not the place for it.
In my view, the goal of the C implementation is a portable library that can be utilised on µCs, bare-metal systems and similar, or be included in compiled C/C++ programs.
From my own perspective, the reason for doing this is to allow small, power-efficient and portable devices to utilise Reticulum for connectivity, and that is my motivation for pursuing this goal. Such devices can include handheld communication devices, but also small integrated Reticulum Transport nodes (imagine a small solar-powered Reticulum node with a LoRa transceiver, that can cover many square kilometers with connectivity). For things as this, we need the C library.
Currently, the only areas where I can see the need for external libraries are the cryptographic primitives needed to assemble Reticulum. I think we can make do with standard C library for the rest. Maybe, maybe not. We'll see, and that's something that needs to be determined and settled on before any code is written.
Other points that need careful thought are:
C or C++? I'd go with C if possible, but I think this might end up being a bit too masochistic an endeavour.
What C standard?
Build system. My personal opinions are quite settled on this, and in favor of gcc and make. Keep the necessary build system to an absolute minimum.
Data/vector structure. This needs careful consideration in regards to dynamic memory limitations.
IO interface abstractions. How is communication with hardware handled? How is communication with host application handled?
Asynchronous IO handling. Threads are probably out of the question here, so an interrupt/timer based approach will need to be carefully designed.
API parity with Python implementation. This is hugely impacted by the choice of either C or C++.
In my opinion, we need clarity on all of these points before any code is written.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
@blademaster98 In the absence of other comments, what would your proposals be for each of your above points? I propose that you make these choices and if you have the time and inclination begin the work. I wonder if Reticulum could end up as a supported protocol of https://github.com/jgromes/RadioLib |
|
Before any of this can start in earnest, the single most important task is to find a well trusted pure C implementation/library that can compile even on memory-limited MCUs, containig the following cryptographic primitives and operations:
|
|
This seems to support most of it Does that seem to be a good place to start? HKDF is not mentioned, but there is this pull request? rweather/arduinolibs@40e2dd2 |
|
@samuk Actually I did some tests days ago. Now I find some thing rather critical for my own purpose of using this protocol missing. For instance, the protocol seems to assume that the route of an established link between two nodes will not change. Which means their is no dynamic routing path mechanism apply, and if one of the transport nodes fails, the message will never be transported to destination. This also can be found in the config file. Mark warns you that you must know what you are doing before you make the key 'enable_transport' true. But for me, in this situation, this is not suitable. So now, I am not sure if I should devote time to do the 'porting to c work'. Sorry. |
|
That is completely understandable @blademaster98. Thanks for your efforts and help so far! Hope ti see you back here when the project is further along. To shed a little more light on the matter, it is true that transport is disabled for peers by default. In a network with many mobile nodes, it is simply counterproductive to the network reliability to have everyone (including very intermittent peers) route traffic. For that case to work, you must allocate a substantial part of the network bandwidth to routing signalling, which is exactly what I have designed Reticulum to avoid, since I wanted it to support very bandwidth limited links. I have not been able to find a methodology that gives you both very low bandwidth and allows fast convergence over intermittent peer routes. And believe me, I have tried. Maybe I am just not smart enough ;) So yes, transport nodes must be enabled by choice in Reticulum based on the two criteria that the node strives to stay up most of the time, and that it stay stationary. I have some ideas to allow moving transport nodes, but that is further down the road. It is however false that there is no dynamic routing path mechanism, but it is totally not obvious how this happens, I must say. The first way this occurs is whenever a peer sends an announce. If the previous best path was broken, the network will converge on the new path as soon as the announce reaches the transport node immediately before the node that dropped off and broke the part. The second, and faster, way is admittedly not completely implemented yet, but it will probably be the most common way for broken paths to be updated once it is completely implemented. When a transport node detects that a previous workable path to a known destination has failed, it simply drops the path table entry and queries the network for a cached announce, which will give it the information it needs to update its path table. The unimplemented part of this is the broken path detection, because I haven't yet found the theoretically correct way to do this. |
|
Hi everybody. Just got to know Reticulum, and it seems a very good candidate for the telecommunication system in HF band I'm working on. Currently we are using UUCP in the transport layer, but Reticulum seems a good modern substitute. I work with C code and can join this C porting endeavor. |
|
Hi Rafael |
|
@rafael2k @markqvist Did you check out my suggestion to use https://github.com/rweather/arduinolibs It supports So a route to a C port might involve
|
|
Hi Samuk, this is a nice to have! I think a C port should be compatible to run either on a multipurpose OS, like Linux, and also in embedded system, as you suggest, in this case, the ATmega MCU. My use is more tailored to bigger systems, specially Linux on ARM, Risc or even x86, but having Reticulum on smaller devices is totally desirable. |
|
Fair enough how about |
|
@samuk Thanks for the suggestion. I think it looks like a good candidate to use for the cryptographic primitives, but I will have to look a little closer at it. But let's just assume for now that we can use it. I don't think we should use the rweather lib as a basis directly, we need to roll the primitives into a library that is portable across many architectures and systems as you mention @rafael2k. Luckily, the rweather lib is under MIT license, so this is no problem. I can do a HKDF and Elliptic Curve Diffie Hellman implementation if needed, but I was hoping to use something already in proven use. |
|
Rweather now has HKDF rweather/arduinolibs@4429613 |
|
That is great news, it's going to speed up things a bit :) |
|
I am also interested in helping here. My primary use for a C port would be to make it easier to use RNS in C programs targeting Linux on x86 and ARM/RISCV. Unfortunately I don't have much experience with embedded systems, but would like to help make that possible as well. Unfortunately, there doesnt seem to be an existing Fernet C implementation so I think I will start by trying to implement that. ^https://github.com/4c3e/fernet-c/tree/main Would love input on choosing the crypto and utility libraries 👍 Currently I'm looking at libsodium and tiny-aes for crypto edit: I'm probably going to spend time working on a python i2p interface implementation before diving in to a complete library rewrite in C 😄 |
|
I'm an electronics engineer, not a software engineer, but I'm watching this very closely. RNS and associated software and hardware make up a brilliant system that I'm very excited about. I was hopeful that a C port would allow RNS to run on a microcontroller system like Arduino in order to minimize power consumption for remote repeater systems. Solar powering an RNode indefinitely would be a lot easier than using a Raspberry Pi (I've been dreaming about updated RNode firmware that would allow it to be a self contained RNS repeater, but I'm not sure that there is enough memory/horsepower in the Atmega). Any thoughts about whether something like this is on the C port road map or if it's even feasible? Edit: |
|
@4c3e and @OtisByron, thanks for joining here and offering your help and interest. The Reticulum C port is getting closer by the day. Currently, there is only one protocol-level addition that needs to be added to the Python implementation, before I am willing to commit to work on the C version (namely an ordered data mode for links). When that is done, and after a round of testing, I think it will be feasible to start the work. @4c3e, if you manage to do a portable C implementation of Fernet, that would help the effort along tremendously. Many of the other components exist in various forms already, some don’t. Have a look at the other comments in this thread, if you didn’t already, there is some good resources regarding crypto libs. Either way, the help is welcome. @OtisByron, regarding the feasibility of Reticulum on very resource-limited hardware like RNode, it is my current position that it is feasible, yes. A device like an RNode will not be able to handle very deep networks with tens of thousands of nodes, due to RAM limitations, but for smaller setups, it should be sufficient as a standalone system. On larger systems with a bit more RAM and CPU, it should also be possible to handle larger networks, but then you also get the higher power consumption, so I really understand where you are coming from with that. One of the design goals of RNode was to use as little power as possible, exactly for purposes like this. I hope this gives a little more clarification about the current status and intents. If you have any questions, ideas or insights, I’d love to hear them. Thanks. |
|
Since this is not an issue per se, but now a pretty long and informative repository on the C-porting effort, I am moving this to a discussion. |
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
A question more than an issue.
Can Reticulum be ported to C? It would benefit all those uC like the ESP32 or the NRF52840 that power many low power LoRa transceivers. Micropython is quite expensive CPU-wise and increases power consumption.
What are the main problems to be faced when porting it to C? I can do it myself but I'll need support throughout the process.
The text was updated successfully, but these errors were encountered: