Update to SDK 1.5.4.1

[devsaurus]

Fixes #1377.

• This PR is compliant with the contributing guidelines (if not, please describe why).
• I have thoroughly tested my contribution.
• The code changes are reflected in the documentation at docs/en/*.

Espressif recently announced a patch to upgrade SDK to 1.5.4.1:

At the first boot up of SDK, the chip internally performs automatic RF calibration according to the conditions of the peripheral components such as the antenna. The RF calibration parameters will be retained in the predetermined sector. A lack of SDK upgrade would decrease the reliability of RF performance. Espressif strongly recommends that users upgrade the SDK to the latest version and install all the patches. The upgrade will not affect any features developed based on the original SDK.

The RF calibration data allocates an additional sector at the end of the flash. I've adjusted the INTERNAL_FLASH_SIZE calculation in cpu_esp8266.h accordingly.

On the downside this most likely breaks 64m and 128m flash support by #1367 since these sizes are not considered by system_get_flash_size_map() and Espressif's recommended new user_main hook user_rf_cal_sector_set. I'd be happy to extend enum flash_size_map in user_interface.h accordingly if somebody confirmed how.

[marcelstoer]

On the downside this most likely breaks 64m and 128m flash support by #1367

@moononournation can we ask you to test this?

[devsaurus]

@moononournation can we ask you to test this?

I'm most curious to learn what system_get_flash_size_map() returns for the 64m and 128m flash chips.

[devsaurus]

@jmattson @raburton I suspect interdependencies between user_rf_cal_sector_set() and your OTA solutions. Please feed back whether defaulting the RF cal sector to end of flash is acceptable.

[devsaurus]

Reverting from 1.5.4.1 back to dev's 1.5.4 gave me some headaches: Although flash download including old init data worked flawlessly, the firmware wasn't able to format spiffs upon initial start-up.
The only solution I found was to erase the whole flash chip with esptool and download again. Guess this purged each and every item that 1.5.4.1 might have placed info flash (with locking/protection?).
Just keep that in mind when test driving the PR.

[pjsg]

I also wonder if there are problems in the spiffs formatting code if there is an odd number of ending sectors used for the firmware. I made the assumption that there were a multiple of 8k in use at the end (which was true). I'll take a look at the code to see if the available space gets rounded down correctly. I would have expected this to cause problems with 1.5.4.1 rather than with the downgrade (unless there are some locked sectors [how do you detect / set these?])

[devsaurus]

I made the assumption that there were a multiple of 8k in use at the end (which was true). I'll take a look at the code to see if the available space gets rounded down correctly.

I wasn't aware of any potential pitfalls here and just assumed that limiting the internal flash size would be enough.

rather than with the downgrade (unless there are some locked sectors [how do you detect / set these?]

Pure speculation from my side as I didn't know about SPIFFS constraints. Technically possible but unlikely IMO.

[pjsg]

There are a couple of places that I need to update to deal with the change of 4 reserved sectors to 5. I'll put up a PR which fixes the code to use the appropriate constant....

Actually, it probably makes more sense to merge the small fix into this branch. I filed a ticket #1399 and the fixed repo is pjsg:correct-sector-count

[marcelstoer]

Reverting from 1.5.4.1 back to dev's 1.5.4 gave me some headaches...The only solution I found was to erase the whole flash chip with esptool and download again.

Do you think we should document this in flash.md (not specifically for this version but a general "How to downgrade")?

[devsaurus]

Thanks for the hint, Marcel. 1.5.4.1 release notes even ask the user to "blank" the new sector. All in all the flash erase seems to be the simplest approach from now on.

BTW SDK 2.0.0 surfaced 😐

[marcelstoer]

This whole "Upgrading from SDK 0.9.x Firmware" chapter starts feeling strange to me. Wouldn't most of what is stated here apply to someone upgrading from e.g. 1.4.x just as well? Besides, if we suggest to erase the flash doesn't separately updating esp_init_data_default.bin become obsolete?

[devsaurus]

Wouldn't most of what is stated here apply to someone upgrading from e.g. 1.4.x just as well?

Thought the same but kept the reference to 0.9.x for the time being since users will get this version on pre-flashed NodeMCU boards. But yes, this chapter could be made more general.

Besides, if we suggest to erase the flash doesn't separately updating esp_init_data_default.bin become obsolete?

Espressif still requests to do so:

Download blank.bin to initialize the sector stored RF_CAL parameter (set by user_rf_cal_sector_set), and download esp_init_data.bin into flash, when the system needs to be initialized, or RF needs to be calibrated again.

Or am I missing something? Honestly, the description in 30d354c still isn't fully clear to me. Does the SDK initialize automatically?

[marcelstoer]

Espressif still requests to do so

Sorry, I also read up on Espressif's release notes - but only after I posted this.

Does the SDK initialize automatically?

I don't know but maybe @jmattsson does?

[jmattsson]

If there is no init data block found during SDK startup, the SDK will install one itself. If there is a previous (potentially too old init block), the SDK probably doesn't do anything with it but there is no documentation from Espressif on this topic.

Doing a chip-erase when upgrading between versions would definitely prevent a mismatch of init data. The alternative is of course to manually overwrite the init data with the new esp_init_data.bin file.

[marcelstoer]

Thanks for confirming what I had thought was going on. I suggest we turn this paragraph into a more general upgrade note with two sections:

• to keep your Lua files: flash NodeMCU and init data (kind of what we currently have)
• if you can give up your Lua files because you're gonna upload them again anyway: erase flash and flash NodeMCU

I can prepare a PR for that if you agree.

[devsaurus]

I can prepare a PR for that if you agree.

I do 😃

BTW does the NodeMCU Flash support erasing? I can't find any trace for that. Seems esptool.py ultimately becomes our reference tool.

[dnc40085]

@marcelstoer @jmattsson I found something when I was checking if SDK 2.0.0 fixed a bug with the forced sleep API.
When I burned the SDK 2.0.0 firmware over the SDK 1.5.4.1 firmware, I didn't update esp_init_data_default.bin and when it rebooted on completion of firmware upload, I got the message system param error, use last saved param! which I assume is the SDK complaining about the old init data, I tracked it to sdk/lib/libmain.a.
Hope that helps.

[pjsg]

I tried a couple of upgrades and downgrades to the 1.5.4.1 version and didn't have any problems. The only issue is that if you have formatted a filesystem on 1.5.4 and then upgrade to 1.5.4.1, the rf cal sector may be part of the file system (which is bad). If you reformat (explicitly) the file system, then it will use the new size.

I don't know if it is worth adding a check to make sure that an existing file system does not violate the space available.

[marcelstoer]

I'm gonna merge this as my manual smoke test suite* runs fine on 1.5.4.1.

However, I anticipate that this update is gonna trip up quite a few of our users because of the flash-erase esp_init_data hassle. It's the first time in roughly a year that updating init data was really necessary. With all previous updates flashing the firmware alone had always worked fine for me.

* 2 different NodeMCU devkits, 1 WeMos D1 mini; different applications using gpio, wifi, tmr, net, http, mqtt, spi modules

[moononournation]

Sorry for the very late reply, I have just ordered WeMos D1 mini Pro and tested with latest firmware. I found the 1.5.4.1 SDK always fall back to use 512 KB flash address for initial rf_cal, but it still can use some workaround to run the NodeMCU firmware correctly. Here are the details:
http://www.instructables.com/id/WeMos-D1-Mini-Pro-Flash-NodeMCU-Firmware/

[marcelstoer]

The current dev should fix this. See #1529 which was merged about a month ago.

[moononournation]

I have tested latest dev branch, it show "rf cal sector: 123" while boot up if connected with 74880 baud rate. Seems 1.5.4.1 SDK not yet define flash size over 4 MB and fall back to 512 KB.