[TW#18642] Changing the partition table (only the coredump) using API

[ammaree]

We have a number of device deployed in the field with the all the usual NVS and OTA data partitions, followed by 3x OTA partitions of equal size.

On the first device deployed we created a 64K "core dump" partition immediately following the #3 OTA partition, with the last 700K or so free.

Since we have completely changed our approach to the configuration and use of core dump we would now like to change the size (and possibly location) of the "core dump" partition programmatically, without having to physically drive to, open and completely reflash each device.

Is this possible, and if so how?

Thanks

[ammaree]

Have had another look at the existing APIs and in theory I am sure it can be done by
a) physically reading the partition table at 0x8000,
b) working through the structure to find the coredump partition entry
c) verifying the details to make sure we have the required spare space both in size and location
d) possibly (optional) to wipe the existing coredump partition space
e) modifying the coredump entry, mainly address and size in the partition table
f) writing the modified partition table back to flash.

Now whilst the above theory might sound simple and straightforward, the practice of implementing this in the light of the complexity of memory mapping and the inter/locking of CPU activity makes it a very risky and complex issue, something I would at this stage prefer to avoid doing myself.

Any chance that this is on the tasklist for somebody, I think it will we of huge value for many people to enable in the field FOTA upgrades on production devices...

All feedback appreciated.

[projectgus]

Hi @ammaree,

Unfortunately, we don't support OTA updating of partition tables. Because the entire 4KB flash sector needs to be erased and then rewritten, there is a short window when a power failure or crash would result in a bricked device requiring physical reflashing via serial. If possible, find a way to work with the legacy partition table you have.

However, if you're sure that a power failure won't occur (and you're not moving at least one of your app partitions) and you're prepared to be very careful then you can rewrite the partition table in place.

Please note that this is entirely unsupported and at your own risk:

1. Configure your app with Writing to Dangerous Regions allowed.
2. Erase sector at 0x8000.
3. Write new partition binary data at 0x8000. You can take the new partition .bin file from the build directory of your app, and follow these steps to embed it as accessible binary data inside the app itself. (Note: the documentation for this function says you can't write data from DROM. This is out of date and I'll update it, you can write from any type of memory.)
4. Reset and let the ESP32 can boot from the new configuration.

Note that any power failure or crash between step 2 & the end of step 3 will probably result in an unbootable device. This window is thankfully only a few tens of milliseconds wide, the exact timing will depending on your flash chip.

Suggest having your app do this directly from app_main, before initialising WiFi or any other tasks, to minimise the chance of unexpected problems. The app can compare the existing partition table with the v1 one in order to determine if it needs to update or not.

Note that changing the partition table can change the offset of one or more other app or data partitions. Your ESP32 will need a working app to boot from after reset - this will be determined by the ota_data partition (if it's moved then you'll need to rewrite it) and the offsets of the app partitions. If you can avoid moving these, then that's by far the easiest approach.

Also note that recent IDF (v3.1-dev) now appends an md5sum to the partition table, and older bootloaders will fail to boot with these partition tables. A backwards compatibility option in menuconfig (to disable the md5sum) should be merged to master this week.

[ammaree]

Hi @projectgus

Thanks for the feedback. For now we are ONLY looking at changing the coredump and want to move it to the end of flash. This will leave the section between 0x00310000 and 0x003CFFFF free for later use.

We would like to use this space for a flash file system, but concerned about future OTA application size needs, hence for now want to leave the space free. This will allow us to expand the OTA partition(s) in future.

Your suggestion to do these changes very early makes sense although our devices are all powered off a large 24V 1000Ah DC battery platform. The result of corrupting the partition table is same as not implementing the functionality, having to go to each site and physically reflash each device, only difference would be that if we do not implement the functionality, the visit to site would be scheduled proactively with less downtime

Q1: We would prefer to not include a full 3K in the application if we can rather build the logic to read, change in ram, erase and write the updated partition back. To do this we would need to recalc an MD5 checksum and rewrite that as well?

Q2: The coming change to disable the MD5 checksum would remove the need to recalc the checksum, but would reduce data integrity checking?

[ammaree]

@projectgus

Thanks for the option to disable the MD5 checksum. Is there any chance for some pointers on manipulating the core dump partition entry directly, structure/rules etc, to avoid sending a whole partition table down?

I also have a separate issue open regarding the size of the actual core dump, any input on that will also be appreciated...

Thanks

[projectgus]

Q1: We would prefer to not include a full 3K in the application if we can rather build the logic to read, change in ram, erase and write the updated partition back. To do this we would need to recalc an MD5 checksum and rewrite that as well?

Sure, this is possible. Also, if you look in a hex editor you'll probably find most of that 3K is 0xFF (ie freshly erased erased bytes) after the end of the end of the actual partition data, so you could always truncate the embedded file to the meaningful length and ignore the rest (after the sector is erased they'll be 0xFF anyhow).

Q2: The coming change to disable the MD5 checksum would remove the need to recalc the checksum, but would reduce data integrity checking?

It does. However the MD5 feature was only added a couple of weeks ago and we've had no reports of partition data corruption in the field. If you read back your "modified" partition table at time of writing and check it, you can probably be fairly sure it's valid.

(Note that writing to the partition table does involve disabling the checks which normally prevent code from writing to "dangerous" regions like the partition table. So there is that...)

The other thing to note is that there isn't anything the bootloader can do if the partition table is invalid, apart from stop booting. So you may prefer for it to try and soldier on, anyhow...

manipulating the core dump partition entry directly, structure/rules etc, to avoid sending a whole partition table down?

Probably the best places to start are:

• http://esp-idf.readthedocs.io/en/latest/api-guides/partition-tables.html
• https://github.com/espressif/esp-idf/blob/master/components/esp32/include/esp_flash_data_types.h#L42

The bootloader mmaps the partition address and treats it as an array of entries to read it. You can do the same:
https://github.com/espressif/esp-idf/blob/master/components/bootloader/subproject/main/bootloader_start.c#L160

(For a non-bootloader app, use the spi_flash_mmap() function - http://esp-idf.readthedocs.io/en/latest/api-reference/storage/spi_flash.html#_CPPv214spi_flash_mmap6size_t6size_t23spi_flash_mmap_memory_tPPKvP23spi_flash_mmap_handle_t )

You can do something like:

• mmap the partition table
• memcpy() from the mapped table to an array in RAM
• Cast to esp_partition_info_t and perform needed changes to the copy in RAM
• Erase parition table sector
• Write data from RAM back to sector
• Verify the new partition table contents

I'd also recommend spending some time looking at the partition table binaries in a hex editor. The structure is pretty simple, so you should be able to get familiar with exactly what is going on in there.

I also have a separate issue open regarding the size of the actual core dump, any input on that will also be appreciated...

I'm probably not the person to ask about that, but I'm sure someone will be able to help.

BTW, in general, for questions rather than ESP-IDF issues the best place to ask is the forum - https://esp32.com

I'm going to close this because I don't think there's an open issue with ESP-IDF here. But let us know (either here on the forum) if you run into any problems modifying partitions.