Partition tables

[mikee47]

Implement class-based API for Sming with support for all architectures.
See #1676 for discussion.

Summary of changes:

• Create new Storage component and port low-level Esp32 partition support
• Use gen_esp32part.py tool as basis for new hwconfig.py tool. Integrate into build system. Add new targets.
• Implement a C++ API. Partition::SubType::App and Partition::SubType::Data are strong enums used to identify all the standard types, but the base API still uses type/subtype so custom types can be accommodated.
• Revise esp8266 sector layout - see updated documentation. This places all the critical information at the start of flash, whereas previously it was at the end. This avoids issues where flash size is set incorrectly.
• Custom sysParam data partition sub-type used for esp8266 system parameters and RF calibration data.
• Add support for 'external' devices (e.g. SPIRAM, supplementary flash, EEPROM, etc.) using a custom Storage partition type. Applications register a Storage::Device object implementation with the Partition API to register these external devices with a C++ interface for them.
• Disentangle SPIFFS from rBoot - OTA upgrading uses partition API.
• Drive SPIFFS image creation from partition table entries - more flexible, multiple images.
• Generate build variables directly from hardware config file (see out/.../hwconfig.mk). Preserve existing build variables where possible, document changes.
• Document hardware configuration with schema and incorporate validation. This can assist with fixing malformed configurations and supplements checks included in the partition tool.
• Add Basic_Storage sample to more thoroughly demonstrate usage
• Update documentation
• Test (HostTests) on all arches
• Remove DISABLE_SPIFFS, SPI_SIZE and SPIFF_SIZE build variables from all samples, including those in submodules.
• Include samples and tests within submodules to CI
• Add check to ensure image size will fit within partition

For other PR:

• Revise rBoot and OtaUpgrade to make more effective use of partitions.
• Update FlashString to support external data storage by Partition?
• Add NVS Component based on or using ESP-IDF for storing configuration data.
• Switch to spiffsgen.py (IFS)

[mikee47]

I dislike the .csv format. It's easy to make mistakes and inflexible. Entries have to be in the correct order.

Instead of this:

# Name,   Type, SubType, Offset,   Size,    Flags
phy_init, data, phy,     0x3000,   0x01000,
nvs,      data, nvs,     0x4000,   0x04000,
app,      app,  factory, 0x008000, 0xf8000,
spiffs,   data, spiffs,  0x100000, 65536,

Add the information to a hardware configuration file:

{
	"Esp8266": {
		"partition-table": {
			"offset": "0x2000",
			"entries": {
				"phy_init": {
					"address": "0x3000",
					"size": "0x1000",
					"type": "data",
					"subtype": "phy",
					"filename": "$(SDK_BASE)/bin/esp_init_data_default.bin"
				},
				"nvs_data": {
					"address": "0x4000",
					"size": "0x4000",
					"type": "data",
					"subtype": "nvs"
				},
				"app_main": {
					"address": "0x8000",
					"size": "0xf8000",
					"type": "app",
					"subtype": "factory",
					"filename": "$(FW_BASE)/rom0.bin"
				},
				"app_0": {
					"address": "0x108000",
					"size": "0xf8000",
					"type": "app",
					"subtype": "ota"
				},
				"spiffs1": {
					"address": "0x200000",
					"size": "0x10000",
					"type": "data",
					"subtype": "spiffs",
					"filename": "$(FW_BASE)/spiff_rom.bin"
				},
				"spiffs2": {
					"comment": "Uninitialised reserve SPIFFS partition",
					"address": "0x21000",
					"size": "0x40000",
					"type": "data",
					"subtype": "spiffs"
				}
			}
		}
	],
	"Esp32": [
		...
	],
	"Host": [
		...
	]
}

It's clearer, multi-arch and we can add other fields as required (e.g. comments).

One file contains all appropriate settings. The build system would generate this for the user application,
which can then be edited as required.

The address is the most important field, so that goes first, but the user can place fields in any order they like.
Parsing this to produce the correct .csv file is trivial, and we can sort the entries if needed.

Generating this from our fixed configuration variables is also much simpler.

The partition tool(s) could probably do with a rewrite...

Additional note: At present the linkage between partitions and content is very indistinct. For example, we generate a SPIFFS image but the image builder doesn't/shouldn't know or care where it's going to be written. (It may have specific offset requirements but that's another matter.) With the JSON format we can connect the image file to the partition.

[mikee47]

Instead of just a partition table configuration file, let's make it part of a 'hardware configuration' file. Post updated.

[mikee47]

For info. I'm currently porting https://github.com/espressif/esp-idf/tree/85cb1e8ca37302d282b041fc5768a2d248e08a32/components/nvs_flash for Sming. It's mainly C++, which is nice :-)

[mikee47]

@slaff The CodeQL is currently failing, probably because it's trying to use python2...

[slaff]

@mikee47 Do you want to add something more before I merge this PR?

[mikee47]

@slaff Few more fixes, but that should be it unless I find anything else.

[mikee47]

@slaff OK, good to go.

[slaff]

@mikee47 Is it possible to change the location of the partition table? I can see that it is possible to set partition_table_offset to my desired location. Example:

{
    "partition_table_offset": "0x3fa000", // <-- HERE
    "name": "SmartGuard V1 Partition Table",
    "base_config": "spiffs",
    "devices": {
        "spiFlash": {
            "size": "4M"
        }
    },

But after that the calculations are not accounting for this. For example I get:

Partition 'rom0' @ 0x00002000-0x000fefff must be located after @ 0x003fb000

And also the map target shows some strange results:

Partition map:
Device            Start       End         Size        Type      SubType   Name              Filename
----------------  ----------  ----------  ----------  --------  --------  ----------------  ------------
spiFlash          0x00000000  0x003f9fff       4072K                      Boot Sector       
spiFlash          0x003fa000  0x003fafff          4K                      Partition Table
...

[mikee47]

@slaff Nope. The partition table offset is adjustable to make more (or less!) space for the bootloader but must be before any defined partitions. This is a requirement for ESP32 so we need to enforce it generally. Is that a problem?

[mikee47]

I'll look at improving the handling of this and adding a note in the documentation.

[slaff]

Is that a problem?

No, not really.... I was hoping to be able to put the partition somewhere else so that I can have bigger roms. With the current state I should decrease the maximum rom size with 4K which can be a problem in the future...

[mikee47]

I guess you're referring to the restricted memory paging capability of the ESP8266?

You've probably already figured that moving phy_init and sys_param means rom0 can move down to 0x3000. Getting that 4K back could be done if we combine the boot sector and partition table. For example:

• rboot.bin: 1712 bytes (typical)
• partitions.bin: 3072 bytes (but 256 bytes would be adequate for most cases, supporting 8 partitions)

So there's easily enough space to squash them together like this. If you feel there's a use case for this then perhaps open an issue and we can take a look at how it might be achieved?

[slaff]

You've probably already figured that moving phy_init and sys_param means rom0 can move down to 0x3000.

Yep.

Getting that 4K back could be done if we combine the boot sector and partition table ... If you feel there's a use case for this then perhaps open an issue

It's ok as it is for now. Thanks for pointing out what is the partition_table_offset for.