OTA for nodemcu #1496
Comments
|
Hello @jmattsson and @raburton, would you mind assisting in this review? |
|
Sure. If you have any specific questions I'll try and answer them. |
|
Thanks @raburton; I've just made a PR in your repo - rebased your code on top of master, it compiles but I didn't yet have a chance to flash it. |
|
Hi @romanchyla
for nodemcu, all PRs are to be made to the dev branch (i.e.: new features are implemented in the dev branch, while master is meant to represent a stable snapshot). Alternatively, you could work on master, but once the feature is ready you will have to port it to the dev branch first before you make the PR. |
|
My own personal feelings about this, without actually having tested the solutions (I've been meaning to do that for a while now):
Advantages of jmattsson's solution:
jmattsson says they are using a minimal bootloader. My takeaway from that is that it's an in-house one (I could be wrong). Actually, whether using rboot is better or not vs. jmattsson's minimal bootloader is something that could be investigated as well. Adding a checksum to that minimal bootloader is probably trivial. If we could get the best of both worlds, it would be awesome, but I would settle for just getting any OTA update functionality formally into the nodemcu firmware. |
|
Not sure how the in-test mode works, but rBoot has a temporary boot mode
which can be used to ensure you don't switch to a ROM that looks good but
doesn't actually work in some functional way. This was added to rBoot after
the integration with nodemcu, but should be easy enough to enable it.
You can share spiffs or use separate ones, that's nothing to do with the
bootloader and entirely a question for the application.
I haven't compared the APIs, but what you say is probably true, I only
added a trivial lua api for rBoot as a demonstration, ideally it would be
improved or replaced by someone more familiar with the internals of nodemcu
and what users actually want to do with it.
|
|
I haven't chimed in here for a while, but currently I'm waiting to see (and play with) the final details of the ESP32 loader/partitioning. There's already OTA support in that loader, but I'm not convinced their partitioning approach is the best when it comes to the OTA features. I'd consider needing a decicated block just to point out which image to use to be more error-prone than having status-bits built into the OTA image itself (the latter being the approach taken in "my" loader). Once I understand more of the ESP32, I'll see what I can come up with that would be suitable across both chips. If @raburton adds ESP32 support to rBoot that would certainly be something I'd look at. Since this is something I'd be doing for $work, there'd likely be a strong preference towards the minimal end of the scale, since we're needing to conserve battery like there's no tomorrow, and spending (m)any unnecessary cycles in a loader is frowned upon. TL;DR: I totally haven't forgotten about getting OTA support into NodeMCU officially, but now waiting on more ESP32 details & hands-on. |
|
Hi @jmattsson, The ESP32 seems really nice and I saw all the work all of you are putting into making nodemcu ready for it, thank you! I'll try my best to help to integrate existing OTA work into the current nodemcu, since I think many people will be using ESP8266 for many months to come (and RTOS branch will be bleeding-edge for some time). If the esp OTA loader then takes over, all will still be good. But maybe rBoot will also evolve to support the ESP32 and then it might be a question "which of them?". I'll try to document all steps so that the decision might be taken quickly. If I understand your comments about unnecessary cycles, you would prefer to have a bootloader that can support both chips (and in extension, if rBoot or your original solution could do that - it would be a very important argument). Can you please or @raburton share your opinion on this subject, once you form it? |
|
I've started by studying the important concepts, it is work in progress: https://github.com/romanchyla/nodemcu-firmware/blob/ota/docs/en/ota.md However, interesting find for @raburton and @jmattsson: Espressif open sourced their bootloader for ESP32 and it has the interesting OTA bits too: https://github.com/espressif/esp-idf/tree/master/components/bootloader/src/main |
|
any more progress? |
|
The ESP32 is an altogether different beast to the ESP8266. I'm happy for the APIs to be different, they are in many other areas already. As for which OTA approach to use for the 8266, I don't want to weigh in too heavily since I can be seen to have a conflict of interest. #806 is field-proven at my $work, but I can see others preferring the expanded boot functionality of rBoot. |
|
@jmattsson of course prefer, but not only. I have working (and production ready!) branch with integrated rBoot, and it works perfect with sdk200 branch too. Unfortunately it have a lot of changes (for build-files too), so I will find a way (and time) to simplify it and prepare a PR. But SDK200 and LWIP is my priorities right now. |
|
Now that #1435 (sdk200) is merged, what is the next step for this? We currently seem to have two field-proven solutions available for OTA updating. I think that at this point either one is good enough. |
|
@devyte now I very busy on my primary work, and so on LWIP branch. But rboot-PR is in my own todo-list. |
|
For what is worth, I'll also get to this soon (a hardware project took all
my time, but it's almost done)
…On Tue, Dec 13, 2016 at 7:13 AM, Yury Popov ***@***.***> wrote:
@devyte <https://github.com/devyte> now I very busy on my primary work,
and so on LWIP branch. But rboot-PR is in my own todo-list.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1496 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAZIkhbPvHRrMqJNihUVA5XrE6Z8en2aks5rHowCgaJpZM4J4T7p>
.
|
|
@romanchyla it's worth a lot :) I actually have 2 projects with development on-hold waiting on this one feature. If I weren't tied up elsewhere, I would have looked into it already. |
|
Has there been any progress at all on this one, or not yet? |
|
It's been five months since last update so I thought I'd ping if there's any progress? dev-esp32 is becoming more and more usable so it'd be great if some kind of OTA update support was there sooner than later. I myself would be happy with a manual firmware update approach where my webserver would receive a file, store it to my filesystem and then flash it to the other partition and at last would mark some flag somewhere to let bootloader start the other partition after next reboot. Simple and fully under control. Then I would be interested in including lua scripts in the firmware update. I think it's possible to attach a SPIFFS to a nodemcu image but I haven't seen it documented. |
|
@joysfera docs/en/spiffs.md documents the existing build infrastructure for pre-build SPIFFS images, FYI. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
Any progress on bringing OTA capabilities to NodeMCU, what with the new node.setpartitiontable() functions and all..? |
Still on my roadmap, but at the moment there's only one developer active on the core work (me) and my focus is getting Lua 53 with its on-ESP LFS building capability. So maybe by early 2020. |
|
At least one complete OTA solution was provided by me nearly 4 years ago (#816) , and other people contributed other options as well. Seems there was insufficient interest (despite plenty of people in these threads) for any of them to get adopted. |
|
Yup, my thinking also. We'd need to gzip compress the image. If you look at our app/uzlib library, I've got a tiny inflator, and the SPIFFS read only library is about 8Kb, so my current thinking is to have a Lua module that wraps a ~16Kb boot loader which will run in the ICACHE region and read and inflate an Anyway, I am up to my eyes in the Lua 5.3 upgrade at the mo, but I hope to get this done before Xmas. If would be really valuable having a collaborator / expert reviewer who can tell me when my approach is 💩 |
|
Will the spiffs and uzlib libraries compile to run on baremetal or do they need to use functions from the esp sdk? Assuming you can get them to compile for baremetal (quite likely, but don't know how much work that would involve without a good look at the code) then you could include that functionality in to a simple bootloader and check/update the app on startup as required. That also assumes you can get enough compression on your rom image to fit it in a spiffs along side a decompressed rom slot. The principal seems sound. Happy to contribute if I can, but like you have many demands on my time! |
Yes
No, just the BOOT ROM API hooks.
Yup, the firmware already has a bootstap startup with a boot-to-boot message passing service for configuration data. The current firmware implements a modified Harvard architecture for the Lua VM so that you can load up to 256Kb of Lua code and constant data into a flash region though 64Kb is enough for a lot of apps and this leaves the ~44Kb heap for data. This bootloader would be somewhere between 16-24Kb so would fit comfortably in the 32Kb ICACHE IRAM0 segment if caching is disabled. I don't understand why you need a decompressed ROM slot as the inflator is a streaming implementation that would read directly from SPIFFS writing page-by-page overwriting the DRAM0, IRAM0 and IROM0 segments. Most of the know-how is available through /espressif/esptool/tree/master/flasher_stub. The whole process is really just a smart flash-to-flash copy. We disable all interrupts and lock out the SDK during this window. |
I've almost got uzlib compiling baremetal, but just trying to work out what the setjmp business is all about, and how to do without it as I don't think that's available.
That's what I'm referring to when I say decompressed rom slot - the normal space taken up by the "installed" rom. If that's over half the size of the rom (hence not being able to do traditional two-rom OTA) and you need a spiffs containing a compressed version of the new rom (and user files), then you'll need a decent compression ratio.
I'm confused about this. It makes it sound like you're replacing the lua rom while it's running, rather than from the bootloader, which is what I thought you were talking about. I thought you intended to download the compressed new rom to the spiffs, reboot and the bootloader would check the spiffs and install the new rom, then boot it (or on a normal boot, note the lack of ota image in the spiffs and just boot the existing rom). |
|
The main advantage of using SPIFFS is that we can use it as a churn FS storing app data, LFS images, or OTA images as needed.
Basically yes that is sort of an option. NodeMCU -- at least on the ESP8266 SDK build -- uses the ROM bootloader, and as we've done in the past we can slot in a user_start trampoline so that when we're on an OTA reboot we never actually enter the SDK or at least that's an option. The way that the SDK 3.0 starts up is different to 1.x and 2.x. It calls But on this path we would never actually start up the Lua VM, rather just drop into the inflate and copy code. Note that this RO SPIFFS code is entirely disjoint from the full featured RW SPIFFS in the firmware. Most of the code is conditionally compiled out so this RO stub is only 6Kb or so (See SPIFFS binary sizing). It is also running with the icache disabled and can therefore be directly layered on top of the BootROM SPIflash API. |
|
Gotcha, that makes sense then. It was the running code from the main rom before the sdk gets going that had me confused - not played with sdk 3. |
|
I will @ you when I post a PR. Thanks for your feedback. 👍 |
|
@raburton's right about the spiffs reading code needing to be integrated with the boot loader in this case though - it can't be part of the application that's about to get overwritten or everything will Really Not Work(tm). The only possible exception is if you hoist all of that code into IRAM and run the upgrade from there, in which case you could conceivably rewrite what you just booted, but that's quite fragile and can leave you with a non-bootable device if there's an interruption during the flash. If the boot loader is responsible for extracting the new image and getting into place, it can be written such that it will retry until it succeeds. Something which I've done for some products in the past (though on Atmel chips) is to have the flash writing functions part of the boot loader (out of necessity), but then also expose them as library functions to the regular app. As in, link in the symbol addresses of the boot loader functions into the app so the app can directly call them. That's probably too fancy for what we're after, but could be considered. |
|
Agreed @jmattsson, having thought a bit more about this overnight I'm back to thinking this must be fully separate (I was starting to think otherwise about |
|
No you aren't using the SDK libraries. Have a look at the Espressif flasher stub that I referenced. It is linked as a standalone stub using its own But this risk is at the 0.001% levels, and if it happens then you are doing no more than you would have had to do anyway. However for the other 99.999% of the time, you have a usable from SPIFFS loader that works. This risk might be unacceptable for professional service provides with a diaspora of ESP devices and here the ESP32 model might be better, but this really needs a minimum 32Mbit Flash config. The bootmode that I discuss is the only one that can work with ESP8285 class configurations. BTW, this risk is no worse than an app using SPIFFS in R/W mode. The SPIFFS code doesn't have any form of transactional recovery on page writes and there are similar powerfail time windows during output to SPIFFS which leaves you with a bricked FS. I am not saying that I've covered all bases, but I have thought about the obvious ones. But let's get Lua 5.3 live first 😊 |
|
I see. Do you have copies of the readonly spiffs and uzlib that will compile baremetal that I can have a play with? |
|
@raburton, I linked to the SPIFFS repo above. Its wiki as linked tells you how to configure it. Read through the pages. But what you are really wanting here is for me to do the few days investigations and configuration that I want to defer until after I've got Lua5.3 out 😄 The copy that we use is in
I also need to validate Peter's indicative sizes on the xtensa toolchain at As to RFC 1951 uses a 32Kb window, so if we want to use standard GZIP then we will need to allocated 32Kb of RAM to this window, so long as your |
Not really, in fact the opposite, I was going to have a look at it for you. But you did say uzlib and spiffs (specifically a cut down readonly version) would compile baremetal so getting hold of these seemed like a good place to start, but the ones you mention in the nodemcu source tree do not.
So you don't have a baremetal compileable spiffs then? The current one is well integrated into nodemcu which is built with the SDK.
Nope, doesn't compile baremetal either, at least not with my compiler, due to a lack of setjmp/longjmp support in the esp rom functions. On the other hand, I can't figure out why the author used these calls, it can probably be done differently. As you previously asserted this was compileable I assumed you'd already been here, hence asking for what you had already done before I duplicated the effort. |
|
@raburton Richard, I didn't mean to imply that the files as-is will compile to bare metal, but more that there aren't any technical impediments and the work needed to get them to compile to bare metal is small (at least compared to the sh*t that I am dealing with at the mo'): getting a baremetal ld file is a process; a fairly straightforward process, but still a process to be gone through nonetheless. The The author of the uzlib routines to all intents and purposes for this argument was me. I took the original LZW code plus some ideas from Paul Sokolovsky (as per attributions in header) but basically did a minimal implementation of the subset of RFC 1951 that we would need for ESP applications. The reason that it uses longjumps is that is the "Lua way" and I am trying to be consistent with the rest of runtime's coding style. Nonetheless, recoding this with goto's to common error exit would be trivial, and looking at the objdump of libc.a, the |
|
I've had a play in between work and written a simple demo bootloader that can read a file from a spiffs, write it out to flash and boot it. Fully baremetal, nothing pulled in from sdk or any other libraries and in about 12.5k (most of which is spiffs) without playing with any options that might optimise that a bit more. I haven't implemented any checking to see if the update is actually new or if it matches the already installed image. Not sure on best way to do this, could say always apply if the file is there in the spiffs (and allow the application to remove it on boot, so it doesn't get rewritten every time) or could CRC the existing rom to see if it matches and only flash if not. Also, not added decompression yet. I don't know much about esp partition tables, not having played with SDK 3 so at the moment the start address for the rom and spiffs are #defined. I wonder if this allows us memory map flash across different 1mb chunks of the flash without having to intercept that mechanism (as rBoot bigflash support did). Or can you set that up in the pre_init bit on boot? |
|
Thanks Richard. I did some rough calcs and it is going to be pretty tight meeting the Sonoff challange (that is fitting in a minimal NodeMCU build with our current firmware and a SPIFFS big enough to hold the new image) unless we gz compress the image. See app/lua/lflash.c for an example of how to call See app/user/user_main.c for how the PT is declared and hooked in. Note that the App can replace this PT via an updated RCR record (see app/platform/platform.c for this implementation). The RCR is just the 1st page of IROM0, and can be written to using flash nand rules for keeping effectively environment variables that can be updated and last from boot to boot, and we can use an RCR to pass the new firmware image filename to the bootstrap. |
Yes, that's what I was thinking when I said you'll need a good compression ratio.
I've incorporated this (with quite a few modifications) into my new bootloader. It's uses a 32k dictionary so you can compress things fine with normal gzip. I've also added comparison of the crc of the image in spiffs against the installed image and if they don't match it will install the new one (after a test extract to ensure it's good) and boot it. If the ota is already installed (or there is no ota in the spiffs) it will just boot the existing rom.
I've not gone this route for my demo, I'd much rather run the bootloader as a separate application and chain load the user application. If you strongly feel you want to do it that way I'm sure you'll be able to take this code and use it with minimal modification. Also, not using this SDK myself, I haven't gone into the partition stuff but I have left a function for getting this info (at the moment returns pre-set values from the header file) which is an easy point to add partition detection if needed.
It was only used it once and, as far as I could tell, it was entirely surplus to requirements. Moving one small block of code allowed it to be eliminated altogether. I spent a while trying to work out if it was doing something I didn't understand, but I really don't think it was (and it seems to work fine after my modification). Code is here: https://github.com/raburton/sboot |
Great. Part of my reason for reworking and rebundling this UZlib library was to make it a simple IoT includable. The Espressif esptool flasher miniz.c is utterly unmaintainable / adaptable IMO. The UZLIB deflator omits the (Sec 3.2.7) Compression with dynamic Huffman codes because this is complex and just isn't doable in ESP8266 resource limits. However the inflator is a full RFC 1951 implementation, so as you suggest using a standard gzip library / utility on the host and the USLIB inflate with a 32Kb dictionary is a good performance compromise. I did some rough benchmarks and using a dynamic encoding scheme gains perhaps an extra 15% compression for firmware image. Not to be sneezed at. The overhead of decompression is more than compensated by the reduced SPIFFS read time as well. The flash erase time is really quite long, so another optimisation that I've done in my LFS loader is to check for a page being all-ones and if so skipping the erase. In practice the page is either all-ones in which case the erase isn't needed or it fails with inequality within a few compares. I had thought about adding a similar optimisation for normal writes so the the page erase / write would be skipped if the existing page already matched. This way we could use the ld file to place the startup trampoline at the head of the image so that it would in practice be a boot invariant and allow powerfail retry. Anyway, thanks for your support, and I'll do a decent review once I've got this Lua 5.3 release out of the way. PS. I took a quite scan. It looks really promising. 😄 Thanks again for the work. |
To give you an idea of a test I ran: image file of 405,477 bytes, compressed to 340,406. Normal startup (mounting spiffs, reading crc and length from .gz file, reading and and calculating crc of the installed rom and starting normal boot) took less than a second. Possibly quite a bit less - I was only timing with my watch! Modifying a byte to requires an update on boot (mounting spiffs, reading crc and length from .gz file, reading and and calculating crc of the installed rom, performing test decompression to ram, erasing sectors, doing real decompression to flash and then booting new image) took about 10 seconds. Didn't break that down into which bits took the time e.g. how long decompression to ram took vs decompression to flash (including the erase), etc. If you think it's the erase that's the worse part then reading first and seeing if it's needed could help. I don't know how likely you are to find a lot of pages that don't need to be erased when replacing an existing image though.
I can see there might be more to gain here. Although I haven't tried binary diffing roms before to see how much they change. If you build new roms from the same sdk they may look pretty similar, but it wouldn't take a big change much to displace everything slightly in the rom and need a full erase & write.
Or just go the separate bootloader route, then you're never trying to update this bit of the flash. Now I see it can take 10 seconds to write the new image that's a much bigger window to have a fault in that'd you'd want to be able to recover from. Simpler and more clearly defined boundaries between function too and should be less work to integrate, just drop some partition reading code into sBoot and away you go. |
|
Separate bootloader route is a good idea, I was using rBoot for last 2 years quite extensively (too lazy to connect rs232) and the failures happen for me quite often. I'm using drag-drop web UI via HTTP to upload firmware directly to flash. Most often failures were:
So far rBoot never let me brick my ESP. HTTP update takes about 11 seconds to upgrade vs 7 over serial which from my perspective is negligible. Also, be careful when compiling NodeMCU at a different address than 0x40210000, I've tried merging rBoot with NodeMCU 3.0.0 with little success I think the firmware partition is hardcoded somewhere at 0x40210000 as it only works if I upload rboot firmware at 0x4020FFF0 or 0x4030FFF0. |
|
@s-pw Pawel, thanks for the comments. The Also as Richard says, this new version will use gzipped images which drop the firmware image sizes by ~ 20%. I hope to integrate this feature around the new year. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
Is it really stale? |
|
Nope |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
Missing feature
OTA - Over the air update of the firmware
Justification
OTA is super important for devices that work in the field. It is not as important for hobbyists, but they might also enjoy little bit of freedom. Even though there are various mechanisms for updating lua scripts, the firmware still remains the same.
There exist two PR #806 and #816 that implement OTA.
I'm proposing to review both, test them and report back to community the results. Hopefully, in the form of PR that could be incorporated.
Requirements
Because of the ssh verify reported not working, I'd like to concentrate on the most recent SDK (so no backward compatibility).
Questions
Here is what I'd like to answer in this Issue.
esp-01 (1Mb version)
esp-12 (32m version; dev kit)
I'll generate detailed questions. PLEASE add whatever concerns you would like to have addressed into the comments. Hopefully, the authors of the respective OTA implementation will be kind enough to help me with some questions/problems.
Also, I'd very interested in having lua snippets that exercise critical functionality. For example, if you know that certain cryptographic functions might be affected, it would be helpful to suggest ways to test them.
The text was updated successfully, but these errors were encountered: