Detecting if hub storage has been written into

[gyenesvi]

I would like to implement saving calibration data and reloading it afterwards. I can see from the docs that I can use hub.system.storage for that purpose and that there are various helpers for conversion of data to and from bytes, seems easy to use. However, I have a problem with detecting whether the storage has been initialized or not, and would like to ask for advice in it.

In general, what I'd like to do is try to read calibration data in the beginning of the program, and if it's already there use it, otherwise do the calibration and save the data. However, as far as I can see the storage contains random data even after installing Pybricks, so I cannot rely on its contents. Ideally, the way I could imagine it to work is that the storage could be reset (for example cleared to zero) when certain actions happen (for example optionally when a new program is uploaded to the hub). That would allow me to at least detect that the storage is empty (assuming that the data I want to store is never all zeros).

Or is there another/better way of detecting whether the storage has already been written into since a certain event?

If not, I think it could be optional to clear the storage upon uploading the program, because sometimes I don't need the storage to be cleared if I am experimenting with other aspects of the program. However, a feature that would be good to have is that if I publish a program, and somebody else loads it on their hub (that maybe had already Pybricks installed), then it should still work correctly. This is why I think loading a new program would be a good place to clear the storage. This could be the default, and could be opted out from by those who understand what they are doing.

So far, the only solution I could come up with is writing a magic number into the first few bytes of the storage, and checking that upon reading.. This way on first startup of the program the storage will not have the magic number with high probability, which signals the need to calibrate and save the data. On next startup, the magic will be there and calibration will not be necessary. Works, but could be improved/simplified.

[laurensvalk]

This is why I think loading a new program would be a good place to clear the storage

Wouldn't you generally want persistent storage... to persist?

Persistent storage can be useful to store things such as color sensor calibration values, precisely so you don't have to calibrate again after loading a program.

So far, the only solution I could come up with is writing a magic number into the first few bytes of the storage

More generally, you could have your data and add some kind of checksum.

However, as far as I can see the storage contains random data even after installing Pybricks

This is probably something we could/should fix.

[jaguilar]

A message authentication code can prove that a message was generated by a particular program. A version number at the front of the message can be used to ensure that the message was written in the same protocol. So personally what I would do is write the message, version number first, then the authentication code / checksum. If the version is not the same as the current version throw it away. If the message authentication code fails, also throw it away.

If you never need to parse multiple versions of your protocol you could make this even simpler by making your protocol version the salt of your checksum/authentication code.

[laurensvalk]

Well I'd say I would like it to persist as long as the same code is running on the hub. When the code changes, there's a chance that the storage contents need to be entirely different.

If it only has to erase while you're connected, you could have a second code tab open with a simple erase program, that you could run with two clicks when needed. This might be simpler than making a special UI for specific use cases, since you can fully choose what the reset should do (maybe write zeros, maybe write default data, etc).

If it has to erase it only sometimes, you could also make your own input method such as a button press to do (or skip) initialization. That's what I did for this robot, so I didn't have to realign the legs every time. (I did not use storage here, but I did use the reset_angle=False option so that it would preserve the angle between runs.)

[gyenesvi]

If it only has to erase while you're connected, you could have a second code tab open with a simple erase program, that you could run with two clicks when needed.

True and I know that; I can even just use the REPL for erasing, and that would work for myself. But what I'd like to achieve is make it work easily for others who are just starting out with Pybricks FW/programs. I build and publish alternate models of C+ sets, and I'd like to put out Pybricks code for controlling them via Xbox controller. Now I find that many people who are just getting into alternate builds are kind of afraid of getting into coding too (and I do sometimes get complaints even about trivial building things and usage of the Control+ app). So as an entry step for them into this world, I'd like to make it dead trivial for the first time: the Pybricks website has nice explanation of installing the FW and running the first program, so ideally they'd only have to go through that and copy-paste and run the code I supply for the build (and the erase would have to happen at this point in time). I would not go into explaining about storage cleaning, that could sound complicated for someone new.. And the program would be using an Xbox controller, so after installation, it would be running without connection to the computer. And this procedure would also work if they had Pybricks installed before because of another build, and had a different program for that that used the storage previously.

This is probably something we could/should fix.

Having the storage cleared to zero when Pybricks is installed would at least solve the cases when a brand new install is used.. Then I could at least instruct them to reinstall Pybricks for a new build. Would be one step better, thanks.

A message authentication code can prove that a message was generated by a particular program.

That sounds like a more complicated version of a magic number. It still needs to use distinct keys for each program to work, which is just the same as using different magic numbers for each program, plus a bunch of complicated cryptography algorithms required for it (sounds like an overkill for writing a few numbers into a storage of capacity of 128 bytes in total). It will also require keeping track of magic numbers / keys somewhere, and people will sooner or later start copying/modifying programs without modifying the magic number/key, and at that point it can break down.

[laurensvalk]

I would not go into explaining about storage cleaning, that could sound complicated for someone new.

Maybe I still don't quite follow. If the goal is to make it easier, why not just initialize steering as we do in our examples, and just like LEGO does in their apps? That way things always work the same, too.

[gyenesvi]

For a simple case as steering it can work that way. But what about more complex calibration, that takes more time or moves the vehicle around and needs special circumstances like a flat surface to start on? It would be annoying to run such a calibration all the time on startup (actually even steering can be problematic, if you run that on uneven surface where the wheels may be blocked, it will just result in a wrongly calibrated range, whereas if you did not run it because it was already calibrated previously it would just work well).

For example, in a current model I managed to implement speed calibration as well. It runs the vehicle forward for some time and measures top speed, to make speed control of drive motors precisely proportional to joystick positions. It needs some space to run, so it would not be a great idea to run it all the time the model is powered on, it could suddenly run off the table for example..

just like LEGO does in their apps

The official C+ profiles don't do so. They only calibrate when the profile is run for the first time, and if you choose to do so manually in the app. This is what I'd like to mimic actually. Calibrate when the program is run first after install, and also provide a way to recalibrate on a button (long) press on the Xbox controller. I got this part working as well, but it's not enough to rely on the button either, because that can already be too late. This is because if a program reads garbage from the storage (and it does not know if the storage contains garbage or not) and uses those values for the init of some objects, those possibly invalid values can crash the program before getting into the main loop and letting the user do a manual calibration through button press.

Altogether I feel like something is conceptually missing here about storage usage, in the sense that the program that runs on the hub may be updated, and the contents of the storage can get 'out of sync' with the current program, and that's not easy to detect. Actually, it would be enough to have a system flag that's reset to zero whenever the program is updated on the hub, and the program could test it and set it to 1 on its first run (maybe could even be set automatically by any program exit). That would be enough for a program to tell if has been run before / if the storage contents are possibly invalid.