From b3a856992cdeba61ca4e2637cda3d26408e90fed Mon Sep 17 00:00:00 2001 From: Slavey Karadzhov Date: Mon, 15 Mar 2021 11:13:36 +0100 Subject: [PATCH 01/16] Initial version. Missing features: - [ ] SSL for authentication and authorization. - [ ] Or firmware signing. - [ ] And a demo script for uploading a new stable and nighly versions. --- samples/Ota_Mqtt/.cproject | 151 +++++++++++++++++++++ samples/Ota_Mqtt/.project | 28 ++++ samples/Ota_Mqtt/Makefile | 9 ++ samples/Ota_Mqtt/README.rst | 49 +++++++ samples/Ota_Mqtt/app/application.cpp | 193 +++++++++++++++++++++++++++ samples/Ota_Mqtt/basic_rboot.hw | 19 +++ samples/Ota_Mqtt/basic_rboot_host.hw | 19 +++ samples/Ota_Mqtt/component.mk | 29 ++++ samples/Ota_Mqtt/files/testfile.txt | 2 + 9 files changed, 499 insertions(+) create mode 100644 samples/Ota_Mqtt/.cproject create mode 100644 samples/Ota_Mqtt/.project create mode 100644 samples/Ota_Mqtt/Makefile create mode 100644 samples/Ota_Mqtt/README.rst create mode 100644 samples/Ota_Mqtt/app/application.cpp create mode 100644 samples/Ota_Mqtt/basic_rboot.hw create mode 100644 samples/Ota_Mqtt/basic_rboot_host.hw create mode 100644 samples/Ota_Mqtt/component.mk create mode 100644 samples/Ota_Mqtt/files/testfile.txt diff --git a/samples/Ota_Mqtt/.cproject b/samples/Ota_Mqtt/.cproject new file mode 100644 index 0000000000..e1450b6031 --- /dev/null +++ b/samples/Ota_Mqtt/.cproject @@ -0,0 +1,151 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + make + -f ${ProjDirPath}/Makefile + all + true + true + true + + + make + -f ${ProjDirPath}/Makefile + clean + true + true + true + + + make + -f ${ProjDirPath}/Makefile + flash + true + true + true + + + make + -f ${ProjDirPath}/Makefile + flashonefile + true + true + true + + + make + -f ${ProjDirPath}/Makefile + flashinit + true + true + true + + + make + -f ${ProjDirPath}/Makefile + flashboot + true + true + true + + + make + -f ${ProjDirPath}/Makefile + rebuild + true + true + true + + + + + + + + + + + + + + + + + + + + diff --git a/samples/Ota_Mqtt/.project b/samples/Ota_Mqtt/.project new file mode 100644 index 0000000000..76f0847014 --- /dev/null +++ b/samples/Ota_Mqtt/.project @@ -0,0 +1,28 @@ + + + Ota_Mqtt + + + SmingFramework + + + + org.eclipse.cdt.managedbuilder.core.genmakebuilder + clean,full,incremental, + + + + + org.eclipse.cdt.managedbuilder.core.ScannerConfigBuilder + full,incremental, + + + + + + org.eclipse.cdt.core.cnature + org.eclipse.cdt.core.ccnature + org.eclipse.cdt.managedbuilder.core.managedBuildNature + org.eclipse.cdt.managedbuilder.core.ScannerConfigNature + + diff --git a/samples/Ota_Mqtt/Makefile b/samples/Ota_Mqtt/Makefile new file mode 100644 index 0000000000..ff51b6c3a7 --- /dev/null +++ b/samples/Ota_Mqtt/Makefile @@ -0,0 +1,9 @@ +##################################################################### +#### Please don't change this file. Use component.mk instead #### +##################################################################### + +ifndef SMING_HOME +$(error SMING_HOME is not set: please configure it as an environment variable) +endif + +include $(SMING_HOME)/project.mk diff --git a/samples/Ota_Mqtt/README.rst b/samples/Ota_Mqtt/README.rst new file mode 100644 index 0000000000..dd9416596d --- /dev/null +++ b/samples/Ota_Mqtt/README.rst @@ -0,0 +1,49 @@ +OTA over MQTT +============= + +.. highlight:: bash + +Introduction +------------ + +This example demonstrates how you can create an application that updates its firmware Over The Air (OTA) using the MQTT protocol. +MTQTT has less overhead compared to HTTP and can be used for faster delivery of application updates. + +Versioning +---------- +To simplify the OTA process and make it less error prone we are using the following versioning principles: +1) For version naming we use `semantic versioning `_. +If our current version is 4.3.1 then 4 is the major, 3 is the minor and 1 is the patch version number. +2) Every application firmware knows its version. +3) An application with the same major and minor version should be compatible for update no matter what the patch number is. +If the new firmware is not compatible then a new minor or major version should be used. + +Theory Of Operation +------------------- +1) On a period of time the application connects to check if there is a new version of the firmware. +In your application this period has to be carefully selected so that OTA updates occur when the device has +enough resources: memory, space on flash, power and time to complete such an update. Also there should be no critical task running at the moment. +Depending on the size of the new firmware and the speed of the connection an update can take 10 to 20 seconds. +2) The application connects via MQTT to a remote server and subscribes to a special topic. The topic is based on the +application id and its current version. If the current application version is 4.3.1 then the topic that will be used for OTA is "/a/test/u/4.3". +3) The application is waiting for new firmware. When the application is on battery than it makes sense to wait for a limited time and if there is no +message coming back to disconnect. + +Firmware packaging +------------------ +The firmware update must come as one MQTT message. The MQTT protocol allows messages with a maximum size of 268435455 bytes approx 260MB. +This should be perfectly enough for a device that has maximum 1MB available for an application ROM. +The message coming from MQTT contains: +- at the start one byte with the patch version of the firmware +- followed by the firmware data itself +For simplicity the patch version is just one byte. This limits us to 256 possible patch versions. +If needed the patch version can be encoded using `varint `_. +In this example there is no encoding, checksum, signature or encryption for the firmware data itself. + +Security +-------- +For additional security a standard SSL/TLS can be used +1) The communication should be secured using standard SSL. +2) To prove that the server is the correct one: The MQTT clients should pin the public key fingerprint on the server. +OR have a list of public key fingerprints that are allowed. +3) To prove that the clients are allowed to connect: Every MQTT client should also have a client certificate that is signed by the server. diff --git a/samples/Ota_Mqtt/app/application.cpp b/samples/Ota_Mqtt/app/application.cpp new file mode 100644 index 0000000000..9f47cf8819 --- /dev/null +++ b/samples/Ota_Mqtt/app/application.cpp @@ -0,0 +1,193 @@ +#include +#include + +// If you want, you can define WiFi settings globally in Eclipse Environment Variables +#ifndef WIFI_SSID +#define WIFI_SSID "PleaseEnterSSID" // Put your SSID and password here +#define WIFI_PWD "PleaseEnterPass" +#endif + +#ifndef APP_VERSION +#include +#define APP_VERSION MACROQUOTE(SMING_MAJOR_VERSION) "." MACROQUOTE(SMING_MINOR_VERSION) +#define APP_VERSION_PATCH SMING_PATCH_VERSION +#endif + +namespace +{ +MqttClient mqtt; + +struct UpdateState { + RbootOutputStream* stream{nullptr}; + bool started{false}; +}; + +Storage::Partition findRomPartition(uint8_t slot) +{ + String name = F("rom"); + name += slot; + auto part = Storage::findPartition(name); + if(!part) { + debug_w("Partition '%s' not found", name.c_str()); + } + return part; +} + +void switchRom() +{ + uint8 before, after; + before = rboot_get_current_rom(); + if(before == 0) { + after = 1; + } else { + after = 0; + } + Serial.printf("Swapping from rom %d to rom %d.\r\n", before, after); + rboot_set_current_rom(after); + Serial.println("Restarting...\r\n"); + System.restart(); +} + +void otaUpdate() +{ + if(mqtt.isProcessing()) { + Serial.println("There is an update in progress. Refusing to start new update."); + return; + } + + uint8 slot; + rboot_config bootconf; + // select rom slot to flash + bootconf = rboot_get_config(); + slot = bootconf.current_rom; + if(slot == 0) { + slot = 1; + } else { + slot = 0; + } + + Serial.println("Checking for a new application firmware..."); + + auto part = findRomPartition(slot); + if(!part) { + Serial.println("FAILED: Cannot find application address"); + return; + } + + mqtt.connect(Url(MQTT_URL), "sming"); + mqtt.setPayloadParser( + [part](MqttPayloadParserState& state, mqtt_message_t* message, const char* buffer, int length) -> int { + if(message == nullptr) { + debug_e("Invalid MQTT message"); + return 1; + } + + if(length == MQTT_PAYLOAD_PARSER_START) { + UpdateState* updateState = new UpdateState(); + updateState->stream = nullptr; + updateState->started = false; + + state.offset = 0; + state.userData = updateState; + return 0; + } + + auto updateState = static_cast(state.userData); + if(updateState == nullptr) { + debug_e("Update failed for unknown reason!"); + return -1; + } + + if(length == MQTT_PAYLOAD_PARSER_END) { + bool skip = (updateState->stream == nullptr); + if(!skip) { + delete updateState->stream; + switchRom(); + } + + return 0; + } + + if(!updateState->started) { + updateState->started = true; + if(message->common.length - 1 > part.size()) { + debug_e("The new rom is too big to fit!"); + return -2; + } + + uint8_t patchVersion = buffer[0]; + if(patchVersion < APP_VERSION_PATCH) { + // The update is not newer than our patch version + return 0; + } + + updateState->started = true; + updateState->stream = new RbootOutputStream(part.address(), part.size()); + buffer++; + length--; + } + + auto rbootStream = static_cast(updateState->stream); + if(rbootStream == nullptr) { + return 0; + } + + auto written = rbootStream->write(reinterpret_cast(buffer), length); + return (written - length); + }); + + String updateTopic = "/a/"; + updateTopic += APP_ID; + updateTopic += "/u/"; + updateTopic += APP_VERSION; + debug_d("Subscribing to topic: %s", updateTopic.c_str()); + mqtt.subscribe(updateTopic); +} + +void showInfo() +{ + Serial.printf("\r\nSDK: v%s\r\n", system_get_sdk_version()); + Serial.printf("Free Heap: %d\r\n", system_get_free_heap_size()); + Serial.printf("CPU Frequency: %d MHz\r\n", system_get_cpu_freq()); + Serial.printf("System Chip ID: %x\r\n", system_get_chip_id()); + + rboot_config conf; + conf = rboot_get_config(); + + debug_d("Count: %d", conf.count); + debug_d("ROM 0: 0x%08x", conf.roms[0]); + debug_d("ROM 1: 0x%08x", conf.roms[1]); + debug_d("ROM 2: 0x%08x", conf.roms[2]); + debug_d("GPIO ROM: %d", conf.gpio_rom); + + Serial.printf("\r\nCurrently running rom %d. Application version: %s\r\n", conf.current_rom, APP_VERSION); + Serial.println(); +} + +void connectOk(IpAddress ip, IpAddress mask, IpAddress gateway) +{ + /* + This application starts the update right after a successful connecton. + In a real-world application you should start the update procedure + only when the chances of success are high enough. + + For example when there is enough power, free RAM and sufficient time + AND there is no critical task running at the moment. + */ + otaUpdate(); +} + +} // end of anonymous namespace + +void init() +{ + Serial.begin(SERIAL_BAUD_RATE); // 115200 by default + Serial.systemDebugOutput(true); // Debug output to serial + + showInfo(); + + WifiStation.enable(true); + WifiStation.config(WIFI_SSID, WIFI_PWD); + + WifiEvents.onStationGotIP(connectOk); +} diff --git a/samples/Ota_Mqtt/basic_rboot.hw b/samples/Ota_Mqtt/basic_rboot.hw new file mode 100644 index 0000000000..64465e1b3a --- /dev/null +++ b/samples/Ota_Mqtt/basic_rboot.hw @@ -0,0 +1,19 @@ +{ + "name": "Two ROM slots, two SPIFFS", + "base_config": "spiffs", + "partitions": { + "rom1": { + "address": "0x108000", + "size": "992K", + "type": "app", + "subtype": "ota_0", + "filename": "$(RBOOT_ROM_1_BIN)" + }, + "spiffs1": { + "address": "0x300000", + "size": "512K", + "type": "data", + "subtype": "spiffs" + } + } +} diff --git a/samples/Ota_Mqtt/basic_rboot_host.hw b/samples/Ota_Mqtt/basic_rboot_host.hw new file mode 100644 index 0000000000..2085d420c1 --- /dev/null +++ b/samples/Ota_Mqtt/basic_rboot_host.hw @@ -0,0 +1,19 @@ +{ + "name": "Two ROM slots, two SPIFFS", + "base_config": "spiffs", + "partitions": { + "rom1": { + "address": "0x108000", + "size": "992K", + "type": "app", + "subtype": "ota_0", + "filename": "$(BLANK_BIN)" + }, + "spiffs1": { + "address": "0x300000", + "size": "512K", + "type": "data", + "subtype": "spiffs" + } + } +} diff --git a/samples/Ota_Mqtt/component.mk b/samples/Ota_Mqtt/component.mk new file mode 100644 index 0000000000..205fc55c40 --- /dev/null +++ b/samples/Ota_Mqtt/component.mk @@ -0,0 +1,29 @@ +## User configurable settings + +# Application id +APP_ID := "test" + +# Application version: string containing only the major and minor version separated by comma +# APP_VERSION := "1.2" +# Application patch version: integer containing only the patch version +# APP_VERSION_PATCH := 3 + +# Firmware Update Server +MQTT_URL := "mqtt://test.mosquitto.org:1883" + +## End of user configurable settings. Don't change anything below this line + +## use rboot build mode +RBOOT_ENABLED := 1 + +## Use standard hardware config with two ROM slots and two SPIFFS partitions +ifeq ($(SMING_ARCH),Host) + HWCONFIG := basic_rboot_host +else + HWCONFIG := basic_rboot +endif + +APP_CFLAGS := -DMQTT_URL="\"$(MQTT_URL)"\" -DAPP_ID="\"$(APP_ID)"\" +ifneq ($(APP_VERSION),) + APP_CFLAGS += -DAPP_VERSION="\"$(APP_VERSION)"\" +endif \ No newline at end of file diff --git a/samples/Ota_Mqtt/files/testfile.txt b/samples/Ota_Mqtt/files/testfile.txt new file mode 100644 index 0000000000..f8b77180ca --- /dev/null +++ b/samples/Ota_Mqtt/files/testfile.txt @@ -0,0 +1,2 @@ +This line is the content of test file. + From 184749ecc0a9371c0e59e699341db3538ef2cc4a Mon Sep 17 00:00:00 2001 From: Slavey Karadzhov Date: Tue, 16 Mar 2021 17:14:55 +0100 Subject: [PATCH 02/16] Added SSL support. --- samples/Ota_Mqtt/app/application.cpp | 23 +++++++++++++++++++++++ samples/Ota_Mqtt/component.mk | 3 +++ 2 files changed, 26 insertions(+) diff --git a/samples/Ota_Mqtt/app/application.cpp b/samples/Ota_Mqtt/app/application.cpp index 9f47cf8819..b8330e2f5e 100644 --- a/samples/Ota_Mqtt/app/application.cpp +++ b/samples/Ota_Mqtt/app/application.cpp @@ -74,6 +74,29 @@ void otaUpdate() return; } + +#ifdef ENABLE_SSL + mqtt.setSslInitHandler([](Ssl::Session& session) { + debug_i("Initialising SSL session for GRC"); + + // Use the Gibson Research fingerprints web page as an example. Unlike Google, these fingerprints change very infrequently. + static const Ssl::Fingerprint::Cert::Sha1 sha1Fingerprint PROGMEM = { + 0xEE, 0xBC, 0x4B, 0xF8, 0x57, 0xE3, 0xD3, 0xE4, 0x07, 0x54, + 0x23, 0x1E, 0xF0, 0xC8, 0xA1, 0x56, 0xE0, 0xD3, 0x1A, 0x1C + }; + + // Trust certificate only if it matches the SHA1 fingerprint... + session.validators.pin(sha1Fingerprint); + + // We're using fingerprints, so don't attempt to validate full certificate + session.options.verifyLater = true; + + // Use all supported cipher suites to make a connection + session.cipherSuites = &Ssl::CipherSuites::full; + }); +#endif + + mqtt.connect(Url(MQTT_URL), "sming"); mqtt.setPayloadParser( [part](MqttPayloadParserState& state, mqtt_message_t* message, const char* buffer, int length) -> int { diff --git a/samples/Ota_Mqtt/component.mk b/samples/Ota_Mqtt/component.mk index 205fc55c40..fe4120318e 100644 --- a/samples/Ota_Mqtt/component.mk +++ b/samples/Ota_Mqtt/component.mk @@ -10,6 +10,9 @@ APP_ID := "test" # Firmware Update Server MQTT_URL := "mqtt://test.mosquitto.org:1883" +ifneq ($(ENABLE_SSL),) + MQTT_URL := "mqtts://test.mosquitto.org:8883" +endif ## End of user configurable settings. Don't change anything below this line From a4448e28c070dd844d0213f8be4e8dbee19fe5d8 Mon Sep 17 00:00:00 2001 From: Slavey Karadzhov Date: Tue, 16 Mar 2021 17:47:44 +0100 Subject: [PATCH 03/16] Added client certificate support. Not tested yet. --- samples/Ota_Mqtt/app/application.cpp | 9 +++++++++ samples/Ota_Mqtt/component.mk | 12 ++++++++++-- samples/Ota_Mqtt/files/certificate.pem.crt.der | Bin 0 -> 787 bytes samples/Ota_Mqtt/files/private.pem.key.der | Bin 0 -> 610 bytes samples/Ota_Mqtt/files/testfile.txt | 2 -- 5 files changed, 19 insertions(+), 4 deletions(-) create mode 100644 samples/Ota_Mqtt/files/certificate.pem.crt.der create mode 100644 samples/Ota_Mqtt/files/private.pem.key.der delete mode 100644 samples/Ota_Mqtt/files/testfile.txt diff --git a/samples/Ota_Mqtt/app/application.cpp b/samples/Ota_Mqtt/app/application.cpp index b8330e2f5e..7adc19a61f 100644 --- a/samples/Ota_Mqtt/app/application.cpp +++ b/samples/Ota_Mqtt/app/application.cpp @@ -17,6 +17,11 @@ namespace { MqttClient mqtt; +#if ENABLE_CLIENT_CERTIFICATE +IMPORT_FSTR(privateKeyData, PROJECT_DIR "/files/private.pem.key.der"); +IMPORT_FSTR(certificateData, PROJECT_DIR "/files/certificate.pem.crt.der"); +#endif + struct UpdateState { RbootOutputStream* stream{nullptr}; bool started{false}; @@ -91,6 +96,10 @@ void otaUpdate() // We're using fingerprints, so don't attempt to validate full certificate session.options.verifyLater = true; +#if ENABLE_CLIENT_CERTIFICATE + session.keyCert.assign(privateKeyData, certificateData); +#endif + // Use all supported cipher suites to make a connection session.cipherSuites = &Ssl::CipherSuites::full; }); diff --git a/samples/Ota_Mqtt/component.mk b/samples/Ota_Mqtt/component.mk index fe4120318e..b9ac8c9d08 100644 --- a/samples/Ota_Mqtt/component.mk +++ b/samples/Ota_Mqtt/component.mk @@ -1,5 +1,9 @@ ## User configurable settings +ENABLE_CLIENT_CERTIFICATE ?= 0 + +CONFIG_VARS := ENABLE_SSL ENABLE_CLIENT_CERTIFICATE + # Application id APP_ID := "test" @@ -11,7 +15,11 @@ APP_ID := "test" # Firmware Update Server MQTT_URL := "mqtt://test.mosquitto.org:1883" ifneq ($(ENABLE_SSL),) - MQTT_URL := "mqtts://test.mosquitto.org:8883" + ifneq ($(ENABLE_CLIENT_CERTIFICATE),0) + MQTT_URL := "mqtts://test.mosquitto.org:8884" + else + MQTT_URL := "mqtts://test.mosquitto.org:8883" + endif endif ## End of user configurable settings. Don't change anything below this line @@ -26,7 +34,7 @@ else HWCONFIG := basic_rboot endif -APP_CFLAGS := -DMQTT_URL="\"$(MQTT_URL)"\" -DAPP_ID="\"$(APP_ID)"\" +APP_CFLAGS = -DMQTT_URL="\"$(MQTT_URL)"\" -DAPP_ID="\"$(APP_ID)"\" -DENABLE_CLIENT_CERTIFICATE=$(ENABLE_CLIENT_CERTIFICATE) ifneq ($(APP_VERSION),) APP_CFLAGS += -DAPP_VERSION="\"$(APP_VERSION)"\" endif \ No newline at end of file diff --git a/samples/Ota_Mqtt/files/certificate.pem.crt.der b/samples/Ota_Mqtt/files/certificate.pem.crt.der new file mode 100644 index 0000000000000000000000000000000000000000..05534491e27661df50383ac69f03327d2df03d2c GIT binary patch literal 787 zcmXqLV&*q!V*I{r6L)ZMACEuxL7e$i z^TmX{iO%IoQw4+4xn94#UY0*WvSFf(>Dz8Q@41%c%TRK0tOlFuIwW7#SJX_x0@Ak!PQNZ`aqo%zqYyyb*Z3yOVzh z>#845d6%^_HOi#4Sm)o0_2{iM-}H26MgRImot@g3GF5NiTBx&LWv$G`?Kvt9^`+0* z3_r);eYGvytlN5;!<^)B<+i(LV;I*ijXt!5ds`Y$az>!HO!MU{`KyE92R<^7Ru7xP zl6!hLmt4=f*`9SXQHyBM%iYO8ixmPGz@9yxv pTYIUX!^28((_7O_N-WBMT-op^p`^8GPsEFt8Z8^&PU^RO1pv937WM!D literal 0 HcmV?d00001 diff --git a/samples/Ota_Mqtt/files/private.pem.key.der b/samples/Ota_Mqtt/files/private.pem.key.der new file mode 100644 index 0000000000000000000000000000000000000000..bfd239930f780cd21038a40aa1b1099da4f7aca9 GIT binary patch literal 610 zcmV-o0-gOZf&yLw0RRGlfdJgw4eJPyi#NduY=`pfy)5Hi4c%|FhUO-RYGkNKVYmnj z$6wCy7X!+J(O|t{Lw9185mILg>*?2aZ;%;)ksLMci#|)8Id`We6LgPWw~y0zZ~sX* z;;A5_J-LOfmts)WQ(Bo<4+jnk5T7g4S zM_vvsz>(m5d2YVJ1=Ydsx9sERPwoK%K>+*giB~+9hYm6igVGpH{o`yzUD#FetH-k( zvS!U;WEl?k!F1uT8l9D;ErnEroI7^XO;GVzPv-cX57c=Ls(}JQ0N`!aZY(pJ%Okr0 zbUKg^Q*~1+#)wPPDLve43v~K|obneImvyH0>K#dDFuoCq8eQ%W*i&kEdDu>HO!uO2 zcLG2@Wlw(d+O&wLjes!0Bv1v+0|MJxE(t~C?(U%1t&fF8SI6p;8#9z-L=6f+?05a5 zXk-oWIM-2ivGJM4fy4m&!JbJ>7!-Jc Date: Tue, 16 Mar 2021 18:02:06 +0100 Subject: [PATCH 04/16] Minor fix for patch version. --- samples/Ota_Mqtt/component.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/samples/Ota_Mqtt/component.mk b/samples/Ota_Mqtt/component.mk index b9ac8c9d08..50185c447c 100644 --- a/samples/Ota_Mqtt/component.mk +++ b/samples/Ota_Mqtt/component.mk @@ -36,5 +36,5 @@ endif APP_CFLAGS = -DMQTT_URL="\"$(MQTT_URL)"\" -DAPP_ID="\"$(APP_ID)"\" -DENABLE_CLIENT_CERTIFICATE=$(ENABLE_CLIENT_CERTIFICATE) ifneq ($(APP_VERSION),) - APP_CFLAGS += -DAPP_VERSION="\"$(APP_VERSION)"\" + APP_CFLAGS += -DAPP_VERSION="\"$(APP_VERSION)"\" -DAPP_VERSION_PATCH=$(APP_VERSION_PATCH) endif \ No newline at end of file From 5b4e85ab84e417ea026b40d3e662fe3884c27f6d Mon Sep 17 00:00:00 2001 From: Slavey Karadzhov Date: Wed, 17 Mar 2021 16:46:47 +0100 Subject: [PATCH 05/16] Coding style fixes. --- samples/Ota_Mqtt/app/application.cpp | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/samples/Ota_Mqtt/app/application.cpp b/samples/Ota_Mqtt/app/application.cpp index 7adc19a61f..5602182459 100644 --- a/samples/Ota_Mqtt/app/application.cpp +++ b/samples/Ota_Mqtt/app/application.cpp @@ -79,16 +79,12 @@ void otaUpdate() return; } - #ifdef ENABLE_SSL mqtt.setSslInitHandler([](Ssl::Session& session) { - debug_i("Initialising SSL session for GRC"); - - // Use the Gibson Research fingerprints web page as an example. Unlike Google, these fingerprints change very infrequently. - static const Ssl::Fingerprint::Cert::Sha1 sha1Fingerprint PROGMEM = { - 0xEE, 0xBC, 0x4B, 0xF8, 0x57, 0xE3, 0xD3, 0xE4, 0x07, 0x54, - 0x23, 0x1E, 0xF0, 0xC8, 0xA1, 0x56, 0xE0, 0xD3, 0x1A, 0x1C - }; + // These fingerprints change very frequently. + static const Ssl::Fingerprint::Cert::Sha1 sha1Fingerprint PROGMEM = {0xEE, 0xBC, 0x4B, 0xF8, 0x57, 0xE3, 0xD3, + 0xE4, 0x07, 0x54, 0x23, 0x1E, 0xF0, 0xC8, + 0xA1, 0x56, 0xE0, 0xD3, 0x1A, 0x1C}; // Trust certificate only if it matches the SHA1 fingerprint... session.validators.pin(sha1Fingerprint); @@ -105,7 +101,6 @@ void otaUpdate() }); #endif - mqtt.connect(Url(MQTT_URL), "sming"); mqtt.setPayloadParser( [part](MqttPayloadParserState& state, mqtt_message_t* message, const char* buffer, int length) -> int { From b66cc084d7a6c70afc74104396c1ec38ffa9fb91 Mon Sep 17 00:00:00 2001 From: Slavey Karadzhov Date: Wed, 17 Mar 2021 17:57:31 +0100 Subject: [PATCH 06/16] Enable support for varint patch version. Useful for nightly build patch versions. --- samples/Ota_Mqtt/README.rst | 56 +++++++++++++++++++++++++ samples/Ota_Mqtt/app/application.cpp | 62 ++++++++++++++++++++++++---- samples/Ota_Mqtt/component.mk | 24 ++++++----- 3 files changed, 123 insertions(+), 19 deletions(-) diff --git a/samples/Ota_Mqtt/README.rst b/samples/Ota_Mqtt/README.rst index dd9416596d..ec2aa86109 100644 --- a/samples/Ota_Mqtt/README.rst +++ b/samples/Ota_Mqtt/README.rst @@ -26,6 +26,9 @@ enough resources: memory, space on flash, power and time to complete such an upd Depending on the size of the new firmware and the speed of the connection an update can take 10 to 20 seconds. 2) The application connects via MQTT to a remote server and subscribes to a special topic. The topic is based on the application id and its current version. If the current application version is 4.3.1 then the topic that will be used for OTA is "/a/test/u/4.3". +2.1) If there is a need to support both stable and unstable/nightly builds then the topic name can have s or u suffix. For example +all stable versions should be published and downloaded from the topic "/a/test/u/4.3/s". For the unstable ones we can use the topic "/a/test/u/4.3/u". +If an application is interested in both then it can subscribe using the following pattern "/a/test/u/4.3/+". 3) The application is waiting for new firmware. When the application is on battery than it makes sense to wait for a limited time and if there is no message coming back to disconnect. @@ -47,3 +50,56 @@ For additional security a standard SSL/TLS can be used 2) To prove that the server is the correct one: The MQTT clients should pin the public key fingerprint on the server. OR have a list of public key fingerprints that are allowed. 3) To prove that the clients are allowed to connect: Every MQTT client should also have a client certificate that is signed by the server. + + +Configuration and Security features +----------------------------------- + +.. envvar:: APP_ID + + Default: "test" + + This variable contains the unique application name. + +.. envvar:: APP_VERSION + + Default: not set + + Contains the application major and minor versions separated by comma. Example "4.2". + If not set will use the current major and minor version from Sming. + +.. envvar::APP_VERSION_PATCH + + Default: not set + + Contains the application patch version as integer. For stable versions you can use 0 until 255. + For unstable versions the current timestamp can be used as a patch version. + +.. envvar:: ENABLE_VARINT_PATCH_VERSION + + Default: 0 (disabled) + + If set to 1 the OTA upgrade mechanism and application will use a `varint `_` + encoding for the patch version. Thus allowing unlimited number of patch versions. Useful for enumerating unstable/nightly releases. + A bit more difficult to read and write but allows for unlimited versions. + + If set to 0 the OTA upgrade mechanism and application will use one byte for the patch version which will limit it to 256 possible patch versions. + Useful for enumarating stable releases. Easier to write and read but limited to 256 versions only. + +.. envvar:: ENABLE_SSL + + Default: unset (disable) + + If set to 1 (highly recommended), OTA upgrade files will be trasnferred securely over TLS/SSL. + +.. envvar:: ENABLE_CLIENT_CERTIFICATE + + Default: 0 (disabled) + + Used in combination with ``ENABLE_SSL``. Set to 1 if the remote server requires the application to authenticate via client certficate. + +.. envvar:: MQTT_URL + + Default: depends on ``ENABLE_SSL`` and ``ENABLE_CLIENT_CERTIFICATE`` values + + Url containing the location of the firmware update MQTT server. diff --git a/samples/Ota_Mqtt/app/application.cpp b/samples/Ota_Mqtt/app/application.cpp index 5602182459..c94503a291 100644 --- a/samples/Ota_Mqtt/app/application.cpp +++ b/samples/Ota_Mqtt/app/application.cpp @@ -25,8 +25,13 @@ IMPORT_FSTR(certificateData, PROJECT_DIR "/files/certificate.pem.crt.der"); struct UpdateState { RbootOutputStream* stream{nullptr}; bool started{false}; + size_t offset{0}; // The bytes used for encoding the version. + size_t version{0}; }; +constexpr const uint8_t VERSION_NOT_READY = -1; +constexpr const uint8_t VERSION_MAX_BYTES_ALLOWED = 24; + Storage::Partition findRomPartition(uint8_t slot) { String name = F("rom"); @@ -53,6 +58,32 @@ void switchRom() System.restart(); } +#if ENABLE_VARINT_PATCH_VERSION +int getPatchVersion(const char* buffer, int length, size_t& offset, size_t versionStart = 0) +{ + size_t version = versionStart; + offset = 0; + int useNextByte = 0; + do { + version += (buffer[offset] & 0x7f); + useNextByte = (buffer[offset++] & 0x80); + } while(useNextByte && (offset < length)); + + if(useNextByte) { + // all the data is consumed and we still don't have a version number?! + return VERSION_NOT_READY; + } + + return version; +} +#else +int getPatchVersion(const char* buffer, int length, size_t& offset, size_t versionStart = 0) +{ + offset = 1; + return buffer[0]; +} +#endif + void otaUpdate() { if(mqtt.isProcessing()) { @@ -136,22 +167,35 @@ void otaUpdate() } if(!updateState->started) { - updateState->started = true; - if(message->common.length - 1 > part.size()) { - debug_e("The new rom is too big to fit!"); - return -2; + size_t offset = 0; + int patchVersion = getPatchVersion(buffer, length, offset, updateState->version); + updateState->offset += offset; +#if ENABLE_VARINT_PATCH_VERSION + if(patchVersion == VERSION_NOT_READY) { + + if(updateState->offset > VERSION_MAX_BYTES_ALLOWED) { + debug_e("Invalid patch version."); + return -3; // + } + return 0; } +#endif - uint8_t patchVersion = buffer[0]; + updateState->started = true; if(patchVersion < APP_VERSION_PATCH) { - // The update is not newer than our patch version + // The update is not newer than our current patch version return 0; } - updateState->started = true; + if(message->common.length - updateState->offset > part.size()) { + debug_e("The new rom is too big to fit!"); + return -2; + } + + length -= offset; + buffer += offset; + updateState->stream = new RbootOutputStream(part.address(), part.size()); - buffer++; - length--; } auto rbootStream = static_cast(updateState->stream); diff --git a/samples/Ota_Mqtt/component.mk b/samples/Ota_Mqtt/component.mk index 50185c447c..2c97a0be95 100644 --- a/samples/Ota_Mqtt/component.mk +++ b/samples/Ota_Mqtt/component.mk @@ -1,11 +1,12 @@ ## User configurable settings ENABLE_CLIENT_CERTIFICATE ?= 0 +ENABLE_VARINT_PATCH_VERSION ?=0 -CONFIG_VARS := ENABLE_SSL ENABLE_CLIENT_CERTIFICATE +CONFIG_VARS := MQTT_URL ENABLE_SSL ENABLE_CLIENT_CERTIFICATE ENABLE_VARINT_PATCH_VERSION # Application id -APP_ID := "test" +APP_ID ?= "test" # Application version: string containing only the major and minor version separated by comma # APP_VERSION := "1.2" @@ -13,13 +14,15 @@ APP_ID := "test" # APP_VERSION_PATCH := 3 # Firmware Update Server -MQTT_URL := "mqtt://test.mosquitto.org:1883" -ifneq ($(ENABLE_SSL),) - ifneq ($(ENABLE_CLIENT_CERTIFICATE),0) - MQTT_URL := "mqtts://test.mosquitto.org:8884" - else - MQTT_URL := "mqtts://test.mosquitto.org:8883" - endif +ifeq ($(MQTT_URL),) + MQTT_URL := "mqtt://test.mosquitto.org:1883" + ifneq ($(ENABLE_SSL),) + ifneq ($(ENABLE_CLIENT_CERTIFICATE),0) + MQTT_URL := "mqtts://test.mosquitto.org:8884" + else + MQTT_URL := "mqtts://test.mosquitto.org:8883" + endif + endif endif ## End of user configurable settings. Don't change anything below this line @@ -34,7 +37,8 @@ else HWCONFIG := basic_rboot endif -APP_CFLAGS = -DMQTT_URL="\"$(MQTT_URL)"\" -DAPP_ID="\"$(APP_ID)"\" -DENABLE_CLIENT_CERTIFICATE=$(ENABLE_CLIENT_CERTIFICATE) +APP_CFLAGS = -DMQTT_URL="\"$(MQTT_URL)"\" -DAPP_ID="\"$(APP_ID)"\" -DENABLE_CLIENT_CERTIFICATE=$(ENABLE_CLIENT_CERTIFICATE) \ + -DENABLE_VARINT_PATCH_VERSION=$(ENABLE_VARINT_PATCH_VERSION) ifneq ($(APP_VERSION),) APP_CFLAGS += -DAPP_VERSION="\"$(APP_VERSION)"\" -DAPP_VERSION_PATCH=$(APP_VERSION_PATCH) endif \ No newline at end of file From f62e8773079f77fe25c6db284b96c648545c2e1a Mon Sep 17 00:00:00 2001 From: Slavey Karadzhov Date: Thu, 18 Mar 2021 13:34:38 +0100 Subject: [PATCH 07/16] Moving the code to a library with some samples. --- Sming/Libraries/OtaUpgradeMqtt/.cs | 0 Sming/Libraries/OtaUpgradeMqtt/README.rst | 85 +++++++++++ Sming/Libraries/OtaUpgradeMqtt/api.rst | 7 + Sming/Libraries/OtaUpgradeMqtt/component.mk | 14 ++ .../OtaUpgradeMqtt/samples/Upgrade}/.cproject | 0 .../OtaUpgradeMqtt/samples/Upgrade/.cs | 0 .../OtaUpgradeMqtt/samples/Upgrade}/.project | 0 .../OtaUpgradeMqtt/samples/Upgrade}/Makefile | 0 .../OtaUpgradeMqtt/samples/Upgrade/README.rst | 82 ++++++++++ .../samples/Upgrade}/app/application.cpp | 143 +++--------------- .../samples/Upgrade}/basic_rboot.hw | 0 .../samples/Upgrade}/basic_rboot_host.hw | 0 .../samples/Upgrade}/component.mk | 24 ++- .../Upgrade}/files/certificate.pem.crt.der | Bin .../Upgrade}/files/private.pem.key.der | Bin Sming/Libraries/OtaUpgradeMqtt/src/.cs | 0 .../src/AdvancedPayloadParser.cpp | 44 ++++++ .../OtaUpgradeMqtt/src/PayloadParser.cpp | 109 +++++++++++++ .../OtaUpgradeMqtt/src/RbootPayloadParser.cpp | 43 ++++++ .../OtaUpgrade/Mqtt/AdvancedPayloadParser.h | 37 +++++ .../include/OtaUpgrade/Mqtt/PayloadParser.h | 75 +++++++++ .../OtaUpgrade/Mqtt/RbootPayloadParser.h | 43 ++++++ samples/Ota_Mqtt/README.rst | 105 ------------- 23 files changed, 576 insertions(+), 235 deletions(-) create mode 100644 Sming/Libraries/OtaUpgradeMqtt/.cs create mode 100644 Sming/Libraries/OtaUpgradeMqtt/README.rst create mode 100644 Sming/Libraries/OtaUpgradeMqtt/api.rst create mode 100644 Sming/Libraries/OtaUpgradeMqtt/component.mk rename {samples/Ota_Mqtt => Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade}/.cproject (100%) create mode 100644 Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/.cs rename {samples/Ota_Mqtt => Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade}/.project (100%) rename {samples/Ota_Mqtt => Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade}/Makefile (100%) create mode 100644 Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/README.rst rename {samples/Ota_Mqtt => Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade}/app/application.cpp (55%) rename {samples/Ota_Mqtt => Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade}/basic_rboot.hw (100%) rename {samples/Ota_Mqtt => Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade}/basic_rboot_host.hw (100%) rename {samples/Ota_Mqtt => Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade}/component.mk (74%) rename {samples/Ota_Mqtt => Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade}/files/certificate.pem.crt.der (100%) rename {samples/Ota_Mqtt => Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade}/files/private.pem.key.der (100%) create mode 100644 Sming/Libraries/OtaUpgradeMqtt/src/.cs create mode 100644 Sming/Libraries/OtaUpgradeMqtt/src/AdvancedPayloadParser.cpp create mode 100644 Sming/Libraries/OtaUpgradeMqtt/src/PayloadParser.cpp create mode 100644 Sming/Libraries/OtaUpgradeMqtt/src/RbootPayloadParser.cpp create mode 100644 Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/AdvancedPayloadParser.h create mode 100644 Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/PayloadParser.h create mode 100644 Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/RbootPayloadParser.h delete mode 100644 samples/Ota_Mqtt/README.rst diff --git a/Sming/Libraries/OtaUpgradeMqtt/.cs b/Sming/Libraries/OtaUpgradeMqtt/.cs new file mode 100644 index 0000000000..e69de29bb2 diff --git a/Sming/Libraries/OtaUpgradeMqtt/README.rst b/Sming/Libraries/OtaUpgradeMqtt/README.rst new file mode 100644 index 0000000000..56ff2e7808 --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/README.rst @@ -0,0 +1,85 @@ +OTA Firmware Upgrade via MQTT +============================= + +.. highlight:: bash + +Introduction +------------ + +This library allows Sming applications to upgrade their firmware Over-The-Air (OTA) using the MQTT protocol. +MTQTT has less overhead compared to HTTP and can be used for faster delivery of application updates. + +Versioning Principles +--------------------- +To simplify the OTA process we strongly recommend the following versioning principles for your application: + +1. Use `semantic versioning `_. + If your current application version is 4.3.1 then 4 is the major, 3 is the minor and 1 is the patch version number. + +2. Every application firmware knows its version. + +3. An application with the same major and minor version should be compatible for update no matter what the patch number is. + If the new firmware is not compatible then a new minor or major version should be used. + +Theory Of Operation +------------------- +1. On a period of time the application connects to check if there is a new version of the firmware. + In your application this period has to be carefully selected so that OTA updates occur when the device has + enough resources: memory, space on flash, power and time to complete such an update. Also there should be no critical task running at the moment. + Depending on the size of the new firmware and the speed of the connection an update can take 10 to 20 seconds. + +2. The application connects via MQTT to a remote server and subscribes to a special topic. The topic is based on the + application id and its current version. If the current application version is 4.3.1 then the topic that will be used for OTA is ``/a/test/u/4.3``. + +3. If there is a need to support both stable and unstable/nightly builds then the topic name can have `s` or `u` suffix. For example + all stable versions should be published and downloaded from the topic ``/a/test/u/4.3/s``. For the unstable ones we can use the topic ``/a/test/u/4.3/u``. + If an application is interested in both then it can subscribe using the following pattern ``/a/test/u/4.3/+``. + +4. The application is waiting for new firmware. When the application is on battery than it makes sense to wait for a limited time and if there is no + message coming back to disconnect. + +Firmware packaging +------------------ +The firmware update must come as one MQTT message. The MQTT protocol allows messages with a maximum size of 268435455 bytes approx 260MB. +This should be perfectly enough for a device that has maximum 1MB available for an application ROM. + +One MQTT message contains: + +- patch version of the firmware +- followed by the firmware data itself + +Based on the :envvar:`ENABLE_OTA_VARINT_VERSION` the patch version can be encoded either using one byte or a `varint `_. +Based on :envvar:`ENABLE_OTA_ADVANCED` the firmware data can be either without any encoding or be signed and encrypted. + +Security +-------- +For additional security a standard SSL/TLS can be used + +1. The communication should be secured using standard SSL. + +2. To prove that the server is the correct one: The MQTT clients should pin the public key fingerprint on the server. + OR have a list of public key fingerprints that are allowed. + +3. To prove that the clients are allowed to connect: Every MQTT client should also have a client certificate that is signed by the server. + +Configuration +------------- + +.. envvar:: ENABLE_OTA_VARINT_VERSION + + Default: 1 (enabled) + + If set to 1 the OTA upgrade mechanism and application will use a `varint `_ + encoding for the patch version. Thus allowing unlimited number of patch versions. Useful for enumerating unstable/nightly releases. + A bit more difficult to read and write but allows for unlimited versions. + + If set to 0 the OTA upgrade mechanism and application will use one byte for the patch version which will limit it to 256 possible patch versions. + Useful for enumarating stable releases. Easier to write and read but limited to 256 versions only. + +.. envvar:: ENABLE_OTA_ADVANCED + + Default: 0 (disabled) + + If set to 1 the library will work with OtaUpgradeStream which supports signature and encryption of the firmware data itself. + See :component:`OtaUpgrade` for details. In the application the AdvancedPayloadParser can be used to do the MQTT message handling. + diff --git a/Sming/Libraries/OtaUpgradeMqtt/api.rst b/Sming/Libraries/OtaUpgradeMqtt/api.rst new file mode 100644 index 0000000000..37505c0733 --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/api.rst @@ -0,0 +1,7 @@ +OTA Upgrade over MQTT classes +============================= + +.. doxygentypedef:: OtaUpgrade::Mqtt + +.. doxygenclass:: OtaUpgrade::Mqtt::PayloadParser +.. doxygenclass:: OtaUpgrade::Mqtt::RbootPayloadParser diff --git a/Sming/Libraries/OtaUpgradeMqtt/component.mk b/Sming/Libraries/OtaUpgradeMqtt/component.mk new file mode 100644 index 0000000000..f7cb72545f --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/component.mk @@ -0,0 +1,14 @@ +COMPONENT_SRCDIRS := src/ +COMPONENT_INCDIRS := src/include + +# If enabled (set to 1) then we can use all sofisticated mechanisms to upgrade the firmware using the ``OtaUpgrade`` library. +ENABLE_OTA_ADVANCED ?= 0 + +# If enabled (set to 1) then we can use unlimited number of patch versions +ENABLE_OTA_VARINT_VERSION ?= 1 + +ifneq ($(ENABLE_OTA_ADVANCED),0) + COMPONENT_DEPENDS := OtaUpgrade +endif + +COMPONENT_VARS := ENABLE_OTA_ADVANCED ENABLE_OTA_VARINT_VERSION \ No newline at end of file diff --git a/samples/Ota_Mqtt/.cproject b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/.cproject similarity index 100% rename from samples/Ota_Mqtt/.cproject rename to Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/.cproject diff --git a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/.cs b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/.cs new file mode 100644 index 0000000000..e69de29bb2 diff --git a/samples/Ota_Mqtt/.project b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/.project similarity index 100% rename from samples/Ota_Mqtt/.project rename to Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/.project diff --git a/samples/Ota_Mqtt/Makefile b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/Makefile similarity index 100% rename from samples/Ota_Mqtt/Makefile rename to Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/Makefile diff --git a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/README.rst b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/README.rst new file mode 100644 index 0000000000..85ab332ee3 --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/README.rst @@ -0,0 +1,82 @@ +OTA over MQTT +============= + +.. highlight:: bash + +Introduction +------------ + +This example demonstrates how you can create an application that updates its firmware Over The Air (OTA) using the MQTT protocol. +This application uses :component:`OtaUpgradeMqtt` and follows the recommended versioning principles. + +Based on :envvar:`ENABLE_OTA_ADVANCED` the firmware data can be either without any encoding or be signed and encrypted. + +Security +-------- +Depending on :envvar:`ENABLE_SSL` a standard SSL/TLS can be enabled. This way + +1. The communication between the application and the server will be encrypted using standard SSL. + +2. To prove that the server is the correct one: The MQTT clients should pin the public key fingerprint on the server. + OR have a list of public key fingerprints that are allowed. + +3. Depending on :envvar:`ENABLE_CLIENT_CERTIFICATE` the application can send a client certificate that is signed by the server. + +Configuration +------------- + +.. envvar:: APP_ID + + Default: "test" + + This variable contains the unique application name. + +.. envvar:: APP_VERSION + + Default: not set + + Contains the application major and minor versions separated by comma. Example "4.2". + If not set will use the current major and minor version from Sming. + +.. envvar::APP_VERSION_PATCH + + Default: not set + + Contains the application patch version as integer. For stable versions you can use 0 until 255. + For unstable versions the current timestamp can be used as a patch version. + +.. envvar:: ENABLE_OTA_VARINT_VERSION + + Default: 1 (enabled) + + If set to 1 the OTA upgrade mechanism and application will use a `varint `_ + encoding for the patch version. Thus allowing unlimited number of patch versions. Useful for enumerating unstable/nightly releases. + A bit more difficult to read and write but allows for unlimited versions. + + If set to 0 the OTA upgrade mechanism and application will use one byte for the patch version which will limit it to 256 possible patch versions. + Useful for enumarating stable releases. Easier to write and read but limited to 256 versions only. + +.. envvar:: ENABLE_OTA_ADVANCED + + Default: 0 (disabled) + + If set to 1 the library will work with OtaUpgradeStream which supports signature and encryption of the firmware data itself. + See :component:`OtaUpgrade` for details. + +.. envvar:: ENABLE_SSL + + Default: unset (disable) + + If set to 1 (highly recommended), OTA upgrade files will be trasnferred securely over TLS/SSL. + +.. envvar:: ENABLE_CLIENT_CERTIFICATE + + Default: 0 (disabled) + + Used in combination with ``ENABLE_SSL``. Set to 1 if the remote server requires the application to authenticate via client certficate. + +.. envvar:: MQTT_URL + + Default: depends on ``ENABLE_SSL`` and ``ENABLE_CLIENT_CERTIFICATE`` values + + Url containing the location of the firmware update MQTT server. diff --git a/samples/Ota_Mqtt/app/application.cpp b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/app/application.cpp similarity index 55% rename from samples/Ota_Mqtt/app/application.cpp rename to Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/app/application.cpp index c94503a291..a8ff6bed73 100644 --- a/samples/Ota_Mqtt/app/application.cpp +++ b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/app/application.cpp @@ -1,5 +1,11 @@ #include -#include +#include +#include +#include + +#if ENABLE_OTA_ADVANCED +#include +#endif // If you want, you can define WiFi settings globally in Eclipse Environment Variables #ifndef WIFI_SSID @@ -22,16 +28,6 @@ IMPORT_FSTR(privateKeyData, PROJECT_DIR "/files/private.pem.key.der"); IMPORT_FSTR(certificateData, PROJECT_DIR "/files/certificate.pem.crt.der"); #endif -struct UpdateState { - RbootOutputStream* stream{nullptr}; - bool started{false}; - size_t offset{0}; // The bytes used for encoding the version. - size_t version{0}; -}; - -constexpr const uint8_t VERSION_NOT_READY = -1; -constexpr const uint8_t VERSION_MAX_BYTES_ALLOWED = 24; - Storage::Partition findRomPartition(uint8_t slot) { String name = F("rom"); @@ -43,47 +39,6 @@ Storage::Partition findRomPartition(uint8_t slot) return part; } -void switchRom() -{ - uint8 before, after; - before = rboot_get_current_rom(); - if(before == 0) { - after = 1; - } else { - after = 0; - } - Serial.printf("Swapping from rom %d to rom %d.\r\n", before, after); - rboot_set_current_rom(after); - Serial.println("Restarting...\r\n"); - System.restart(); -} - -#if ENABLE_VARINT_PATCH_VERSION -int getPatchVersion(const char* buffer, int length, size_t& offset, size_t versionStart = 0) -{ - size_t version = versionStart; - offset = 0; - int useNextByte = 0; - do { - version += (buffer[offset] & 0x7f); - useNextByte = (buffer[offset++] & 0x80); - } while(useNextByte && (offset < length)); - - if(useNextByte) { - // all the data is consumed and we still don't have a version number?! - return VERSION_NOT_READY; - } - - return version; -} -#else -int getPatchVersion(const char* buffer, int length, size_t& offset, size_t versionStart = 0) -{ - offset = 1; - return buffer[0]; -} -#endif - void otaUpdate() { if(mqtt.isProcessing()) { @@ -133,79 +88,23 @@ void otaUpdate() #endif mqtt.connect(Url(MQTT_URL), "sming"); - mqtt.setPayloadParser( - [part](MqttPayloadParserState& state, mqtt_message_t* message, const char* buffer, int length) -> int { - if(message == nullptr) { - debug_e("Invalid MQTT message"); - return 1; - } - - if(length == MQTT_PAYLOAD_PARSER_START) { - UpdateState* updateState = new UpdateState(); - updateState->stream = nullptr; - updateState->started = false; - - state.offset = 0; - state.userData = updateState; - return 0; - } - - auto updateState = static_cast(state.userData); - if(updateState == nullptr) { - debug_e("Update failed for unknown reason!"); - return -1; - } - - if(length == MQTT_PAYLOAD_PARSER_END) { - bool skip = (updateState->stream == nullptr); - if(!skip) { - delete updateState->stream; - switchRom(); - } - - return 0; - } - if(!updateState->started) { - size_t offset = 0; - int patchVersion = getPatchVersion(buffer, length, offset, updateState->version); - updateState->offset += offset; -#if ENABLE_VARINT_PATCH_VERSION - if(patchVersion == VERSION_NOT_READY) { - - if(updateState->offset > VERSION_MAX_BYTES_ALLOWED) { - debug_e("Invalid patch version."); - return -3; // - } - return 0; - } +#if ENABLE_OTA_ADVANCED + /** + * The advanced parser suppors all firmware upgrades supported by the `OtaUpgrade` library. + * It comes with firmware signing, firmware encryption and so on. + */ + auto parser = new OtaUpgrade::Mqtt::AdvancedPayloadParser(APP_VERSION_PATCH); +#else + /** + * The command below uses class that stores the firmware directly + * using RbootOutputStream on a location provided by us + */ + auto parser = new OtaUpgrade::Mqtt::RbootPayloadParser(part, APP_VERSION_PATCH); #endif - updateState->started = true; - if(patchVersion < APP_VERSION_PATCH) { - // The update is not newer than our current patch version - return 0; - } - - if(message->common.length - updateState->offset > part.size()) { - debug_e("The new rom is too big to fit!"); - return -2; - } - - length -= offset; - buffer += offset; - - updateState->stream = new RbootOutputStream(part.address(), part.size()); - } - - auto rbootStream = static_cast(updateState->stream); - if(rbootStream == nullptr) { - return 0; - } - - auto written = rbootStream->write(reinterpret_cast(buffer), length); - return (written - length); - }); + mqtt.setPayloadParser([parser](MqttPayloadParserState& state, mqtt_message_t* message, const char* buffer, + int length) -> int { return parser->parse(state, message, buffer, length); }); String updateTopic = "/a/"; updateTopic += APP_ID; diff --git a/samples/Ota_Mqtt/basic_rboot.hw b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/basic_rboot.hw similarity index 100% rename from samples/Ota_Mqtt/basic_rboot.hw rename to Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/basic_rboot.hw diff --git a/samples/Ota_Mqtt/basic_rboot_host.hw b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/basic_rboot_host.hw similarity index 100% rename from samples/Ota_Mqtt/basic_rboot_host.hw rename to Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/basic_rboot_host.hw diff --git a/samples/Ota_Mqtt/component.mk b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/component.mk similarity index 74% rename from samples/Ota_Mqtt/component.mk rename to Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/component.mk index 2c97a0be95..9f6d6b03da 100644 --- a/samples/Ota_Mqtt/component.mk +++ b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/component.mk @@ -1,10 +1,6 @@ ## User configurable settings -ENABLE_CLIENT_CERTIFICATE ?= 0 -ENABLE_VARINT_PATCH_VERSION ?=0 - -CONFIG_VARS := MQTT_URL ENABLE_SSL ENABLE_CLIENT_CERTIFICATE ENABLE_VARINT_PATCH_VERSION - +## [Application id and version] ## # Application id APP_ID ?= "test" @@ -13,7 +9,15 @@ APP_ID ?= "test" # Application patch version: integer containing only the patch version # APP_VERSION_PATCH := 3 -# Firmware Update Server +## [TLS/SSL settings ] ## +# Uncomment the line below to start using SSL +# ENABLE_SSL := Bearssl + +# Set this to one if the remote firmware server requires client certificate +# This option is in effect only when ENABLE_SSL is set +ENABLE_CLIENT_CERTIFICATE ?= 0 + +## [ Firmware Update Server ] ## ifeq ($(MQTT_URL),) MQTT_URL := "mqtt://test.mosquitto.org:1883" ifneq ($(ENABLE_SSL),) @@ -25,8 +29,12 @@ ifeq ($(MQTT_URL),) endif endif + ## End of user configurable settings. Don't change anything below this line +CONFIG_VARS := MQTT_URL ENABLE_SSL ENABLE_CLIENT_CERTIFICATE ENABLE_OTA_ADVANCED +COMPONENT_DEPENDS := OtaUpgradeMqtt + ## use rboot build mode RBOOT_ENABLED := 1 @@ -38,7 +46,7 @@ else endif APP_CFLAGS = -DMQTT_URL="\"$(MQTT_URL)"\" -DAPP_ID="\"$(APP_ID)"\" -DENABLE_CLIENT_CERTIFICATE=$(ENABLE_CLIENT_CERTIFICATE) \ - -DENABLE_VARINT_PATCH_VERSION=$(ENABLE_VARINT_PATCH_VERSION) + -DENABLE_OTA_ADVANCED=$(ENABLE_OTA_ADVANCED) ifneq ($(APP_VERSION),) APP_CFLAGS += -DAPP_VERSION="\"$(APP_VERSION)"\" -DAPP_VERSION_PATCH=$(APP_VERSION_PATCH) -endif \ No newline at end of file +endif \ No newline at end of file diff --git a/samples/Ota_Mqtt/files/certificate.pem.crt.der b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/files/certificate.pem.crt.der similarity index 100% rename from samples/Ota_Mqtt/files/certificate.pem.crt.der rename to Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/files/certificate.pem.crt.der diff --git a/samples/Ota_Mqtt/files/private.pem.key.der b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/files/private.pem.key.der similarity index 100% rename from samples/Ota_Mqtt/files/private.pem.key.der rename to Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/files/private.pem.key.der diff --git a/Sming/Libraries/OtaUpgradeMqtt/src/.cs b/Sming/Libraries/OtaUpgradeMqtt/src/.cs new file mode 100644 index 0000000000..e69de29bb2 diff --git a/Sming/Libraries/OtaUpgradeMqtt/src/AdvancedPayloadParser.cpp b/Sming/Libraries/OtaUpgradeMqtt/src/AdvancedPayloadParser.cpp new file mode 100644 index 0000000000..6338c780b0 --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/src/AdvancedPayloadParser.cpp @@ -0,0 +1,44 @@ +/**** + * Sming Framework Project - Open Source framework for high efficiency native ESP8266 development. + * Created 2015 by Skurydin Alexey + * http://github.com/SmingHub/Sming + * All files of the Sming Core are provided under the LGPL v3 license. + * + * AdvancedPayloadParser.h + * + * Created: 2021 - Slavey Karadzhov + * + ****/ + +#include "include/OtaUpgrade/Mqtt/AdvancedPayloadParser.h" + +namespace OtaUpgrade +{ +namespace Mqtt +{ +bool AdvancedPayloadParser::switchRom(const UpdateState& updateState) +{ + if(updateState.stream == nullptr) { + return false; + } + + auto otaStream = static_cast(updateState.stream); + if(otaStream == nullptr) { + return false; + } + + if(otaStream->hasError()) { + debug_e("Got error: %s", toString(otaStream->errorCode).c_str()); + return false; + } + + return true; +} + +ReadWriteStream* AdvancedPayloadParser::getStorageStream(size_t storageSize) +{ + return new OtaUpgradeStream(); +} + +} // namespace Mqtt +} // namespace OtaUpgrade diff --git a/Sming/Libraries/OtaUpgradeMqtt/src/PayloadParser.cpp b/Sming/Libraries/OtaUpgradeMqtt/src/PayloadParser.cpp new file mode 100644 index 0000000000..742efdad6e --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/src/PayloadParser.cpp @@ -0,0 +1,109 @@ +#include "include/OtaUpgrade/Mqtt/PayloadParser.h" + +namespace OtaUpgrade +{ +namespace Mqtt +{ +int PayloadParser::parse(MqttPayloadParserState& state, mqtt_message_t* message, const char* buffer, int length) +{ + if(message == nullptr) { + debug_e("Invalid MQTT message"); + return 1; + } + + if(length == MQTT_PAYLOAD_PARSER_START) { + UpdateState* updateState = new UpdateState(); + updateState->stream = nullptr; + updateState->started = false; + + state.offset = 0; + state.userData = updateState; + return 0; + } + + auto updateState = static_cast(state.userData); + if(updateState == nullptr) { + debug_e("Update failed for unknown reason!"); + return -1; + } + + if(length == MQTT_PAYLOAD_PARSER_END) { + bool skip = (updateState->stream == nullptr); + if(!skip) { + bool success = switchRom(*updateState); + delete updateState->stream; + delete updateState; + if(success) { + debug_d("Swtiching was successful. Restarting..."); + System.restart(); + } else { + debug_e("Swtiching failed!"); + } + } + + return 0; + } + + if(!updateState->started) { + size_t offset = 0; + int patchVersion = getPatchVersion(buffer, length, offset, updateState->version); + state.offset += offset; +#if ENABLE_OTA_VARINT_VERSION + if(currentPatchVersion < 0) { + if(state.offset > VERSION_MAX_BYTES_ALLOWED) { + debug_e("Invalid patch version."); + return -3; // + } + return 0; + } +#endif + + updateState->started = true; + if(size_t(patchVersion) < currentPatchVersion) { + // The update is not newer than our current patch version + return 0; + } + + length -= offset; + buffer += offset; + + updateState->stream = getStorageStream(message->common.length - offset); + } + + auto stream = updateState->stream; + if(stream == nullptr) { + return 0; + } + + auto written = stream->write(reinterpret_cast(buffer), length); + return (written - length); +} + +#if ENABLE_OTA_VARINT_VERSION +int PayloadParser::getPatchVersion(const char* buffer, int length, size_t& offset, size_t versionStart) +{ + size_t version = versionStart; + offset = 0; + int useNextByte = 0; + do { + version += (buffer[offset] & 0x7f); + useNextByte = (buffer[offset++] & 0x80); + } while(useNextByte && (offset < length)); + + if(useNextByte) { + // all the data is consumed and we still don't have a version number?! + return VERSION_NOT_READY; + } + + return version; +} +#else +int PayloadParser::getPatchVersion(const char* buffer, int length, size_t& offset, size_t versionStart) +{ + offset = 1; + return buffer[0]; +} +#endif + +} // namespace Mqtt +} // namespace OtaUpgrade diff --git a/Sming/Libraries/OtaUpgradeMqtt/src/RbootPayloadParser.cpp b/Sming/Libraries/OtaUpgradeMqtt/src/RbootPayloadParser.cpp new file mode 100644 index 0000000000..e2a91de07b --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/src/RbootPayloadParser.cpp @@ -0,0 +1,43 @@ +/**** + * Sming Framework Project - Open Source framework for high efficiency native ESP8266 development. + * Created 2015 by Skurydin Alexey + * http://github.com/SmingHub/Sming + * All files of the Sming Core are provided under the LGPL v3 license. + * + * PayloadParser.h + * + * Created: 2021 - Slavey Karadzhov + * + ****/ + +#include "include/OtaUpgrade/Mqtt/RbootPayloadParser.h" + +namespace OtaUpgrade +{ +namespace Mqtt +{ +bool RbootPayloadParser::switchRom(const UpdateState& updateState) +{ + uint8 before, after; + before = rboot_get_current_rom(); + if(before == 0) { + after = 1; + } else { + after = 0; + } + debug_d("Swapping from rom %d to rom %d.\r\n", before, after); + return rboot_set_current_rom(after); +} + +ReadWriteStream* RbootPayloadParser::getStorageStream(size_t storageSize) +{ + if(storageSize > part.size()) { + debug_e("The new rom is too big to fit!"); + return nullptr; + } + + return new RbootOutputStream(part.address(), part.size()); +} + +} // namespace Mqtt +} // namespace OtaUpgrade diff --git a/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/AdvancedPayloadParser.h b/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/AdvancedPayloadParser.h new file mode 100644 index 0000000000..1cf01191cc --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/AdvancedPayloadParser.h @@ -0,0 +1,37 @@ +/**** + * Sming Framework Project - Open Source framework for high efficiency native ESP8266 development. + * Created 2015 by Skurydin Alexey + * http://github.com/SmingHub/Sming + * All files of the Sming Core are provided under the LGPL v3 license. + * + * AdvancedPayloadParser.h + * + * Created: 2021 - Slavey Karadzhov + * + ****/ + +#pragma once + +#include "PayloadParser.h" +#include + +namespace OtaUpgrade +{ +namespace Mqtt +{ +/** + * This parser allows the processing of firmware data + * that can be encrypted or have signature + */ +class AdvancedPayloadParser : public PayloadParser +{ +public: + using PayloadParser::PayloadParser; + + bool switchRom(const UpdateState& updateState) override; + + ReadWriteStream* getStorageStream(size_t storageSize) override; +}; + +} // namespace Mqtt +} // namespace OtaUpgrade diff --git a/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/PayloadParser.h b/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/PayloadParser.h new file mode 100644 index 0000000000..88ee13c952 --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/PayloadParser.h @@ -0,0 +1,75 @@ +/**** + * Sming Framework Project - Open Source framework for high efficiency native ESP8266 development. + * Created 2015 by Skurydin Alexey + * http://github.com/SmingHub/Sming + * All files of the Sming Core are provided under the LGPL v3 license. + * + * PayloadParser.h + * + * Created: 2021 - Slavey Karadzhov + * + ****/ + +#pragma once + +#include +#include +#include + +namespace OtaUpgrade +{ +namespace Mqtt +{ +constexpr const uint8_t VERSION_NOT_READY = -1; +constexpr const uint8_t VERSION_MAX_BYTES_ALLOWED = 24; + +class PayloadParser +{ +public: + struct UpdateState { + ReadWriteStream* stream{nullptr}; + bool started{false}; + size_t version{0}; + }; + + PayloadParser(size_t currentPatchVersion) : currentPatchVersion(currentPatchVersion) + { + } + + virtual ~PayloadParser() + { + } + + /** + * @brief This method is responsible for switching the rom. + * This method is NOT restarting the system. It will happen lated in the parse method + * @retval true if the switch was successful + */ + virtual bool switchRom(const UpdateState& updateState) = 0; + + /** + * @brief Creates new stream to store the firmware update + * @param size_t storageSize the requested storage size + * + * @retval ReadWriteStream* + */ + virtual ReadWriteStream* getStorageStream(size_t storageSize) = 0; + + /** + * @brief This method takes care to read the incoming MQTT message and pass it to the stream that + * is responsoble for storing the data. + * + * @retval int + * 0 when everything is ok + * <0 when an error has occurred + */ + int parse(MqttPayloadParserState& state, mqtt_message_t* message, const char* buffer, int length); + +private: + size_t currentPatchVersion; + + int getPatchVersion(const char* buffer, int length, size_t& offset, size_t versionStart = 0); +}; + +} // namespace Mqtt +} // namespace OtaUpgrade diff --git a/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/RbootPayloadParser.h b/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/RbootPayloadParser.h new file mode 100644 index 0000000000..0289627d62 --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/RbootPayloadParser.h @@ -0,0 +1,43 @@ +/**** + * Sming Framework Project - Open Source framework for high efficiency native ESP8266 development. + * Created 2015 by Skurydin Alexey + * http://github.com/SmingHub/Sming + * All files of the Sming Core are provided under the LGPL v3 license. + * + * PayloadParser.h + * + * Created: 2021 - Slavey Karadzhov + * + ****/ + +#pragma once + +#include "PayloadParser.h" +#include + +namespace OtaUpgrade +{ +namespace Mqtt +{ +/** + * @brief This parser allows the processing of firmware data that is directly stored + * to the flash memory using RbootOutputStream. + */ +class RbootPayloadParser : public PayloadParser +{ +public: + RbootPayloadParser(const Storage::Partition& part, size_t currentVersion) + : PayloadParser(currentVersion), part(part) + { + } + + bool switchRom(const UpdateState& updateState) override; + + ReadWriteStream* getStorageStream(size_t storageSize) override; + +private: + Storage::Partition part; +}; + +} // namespace Mqtt +} // namespace OtaUpgrade diff --git a/samples/Ota_Mqtt/README.rst b/samples/Ota_Mqtt/README.rst deleted file mode 100644 index ec2aa86109..0000000000 --- a/samples/Ota_Mqtt/README.rst +++ /dev/null @@ -1,105 +0,0 @@ -OTA over MQTT -============= - -.. highlight:: bash - -Introduction ------------- - -This example demonstrates how you can create an application that updates its firmware Over The Air (OTA) using the MQTT protocol. -MTQTT has less overhead compared to HTTP and can be used for faster delivery of application updates. - -Versioning ----------- -To simplify the OTA process and make it less error prone we are using the following versioning principles: -1) For version naming we use `semantic versioning `_. -If our current version is 4.3.1 then 4 is the major, 3 is the minor and 1 is the patch version number. -2) Every application firmware knows its version. -3) An application with the same major and minor version should be compatible for update no matter what the patch number is. -If the new firmware is not compatible then a new minor or major version should be used. - -Theory Of Operation -------------------- -1) On a period of time the application connects to check if there is a new version of the firmware. -In your application this period has to be carefully selected so that OTA updates occur when the device has -enough resources: memory, space on flash, power and time to complete such an update. Also there should be no critical task running at the moment. -Depending on the size of the new firmware and the speed of the connection an update can take 10 to 20 seconds. -2) The application connects via MQTT to a remote server and subscribes to a special topic. The topic is based on the -application id and its current version. If the current application version is 4.3.1 then the topic that will be used for OTA is "/a/test/u/4.3". -2.1) If there is a need to support both stable and unstable/nightly builds then the topic name can have s or u suffix. For example -all stable versions should be published and downloaded from the topic "/a/test/u/4.3/s". For the unstable ones we can use the topic "/a/test/u/4.3/u". -If an application is interested in both then it can subscribe using the following pattern "/a/test/u/4.3/+". -3) The application is waiting for new firmware. When the application is on battery than it makes sense to wait for a limited time and if there is no -message coming back to disconnect. - -Firmware packaging ------------------- -The firmware update must come as one MQTT message. The MQTT protocol allows messages with a maximum size of 268435455 bytes approx 260MB. -This should be perfectly enough for a device that has maximum 1MB available for an application ROM. -The message coming from MQTT contains: -- at the start one byte with the patch version of the firmware -- followed by the firmware data itself -For simplicity the patch version is just one byte. This limits us to 256 possible patch versions. -If needed the patch version can be encoded using `varint `_. -In this example there is no encoding, checksum, signature or encryption for the firmware data itself. - -Security --------- -For additional security a standard SSL/TLS can be used -1) The communication should be secured using standard SSL. -2) To prove that the server is the correct one: The MQTT clients should pin the public key fingerprint on the server. -OR have a list of public key fingerprints that are allowed. -3) To prove that the clients are allowed to connect: Every MQTT client should also have a client certificate that is signed by the server. - - -Configuration and Security features ------------------------------------ - -.. envvar:: APP_ID - - Default: "test" - - This variable contains the unique application name. - -.. envvar:: APP_VERSION - - Default: not set - - Contains the application major and minor versions separated by comma. Example "4.2". - If not set will use the current major and minor version from Sming. - -.. envvar::APP_VERSION_PATCH - - Default: not set - - Contains the application patch version as integer. For stable versions you can use 0 until 255. - For unstable versions the current timestamp can be used as a patch version. - -.. envvar:: ENABLE_VARINT_PATCH_VERSION - - Default: 0 (disabled) - - If set to 1 the OTA upgrade mechanism and application will use a `varint `_` - encoding for the patch version. Thus allowing unlimited number of patch versions. Useful for enumerating unstable/nightly releases. - A bit more difficult to read and write but allows for unlimited versions. - - If set to 0 the OTA upgrade mechanism and application will use one byte for the patch version which will limit it to 256 possible patch versions. - Useful for enumarating stable releases. Easier to write and read but limited to 256 versions only. - -.. envvar:: ENABLE_SSL - - Default: unset (disable) - - If set to 1 (highly recommended), OTA upgrade files will be trasnferred securely over TLS/SSL. - -.. envvar:: ENABLE_CLIENT_CERTIFICATE - - Default: 0 (disabled) - - Used in combination with ``ENABLE_SSL``. Set to 1 if the remote server requires the application to authenticate via client certficate. - -.. envvar:: MQTT_URL - - Default: depends on ``ENABLE_SSL`` and ``ENABLE_CLIENT_CERTIFICATE`` values - - Url containing the location of the firmware update MQTT server. From 38ae4383e2a765ea767d612007fbdce3c70debcb Mon Sep 17 00:00:00 2001 From: Slavey Karadzhov Date: Thu, 18 Mar 2021 17:59:57 +0100 Subject: [PATCH 08/16] Add MQTT also to the list of supported OTA transports. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 5a0f819509..2676f91342 100644 --- a/README.md +++ b/README.md @@ -68,7 +68,7 @@ Please note Version 4 documentation is at [sming.readthedocs.io](https://sming.r * With clients supporting: HTTP, MQTT, WebSockets and SMTP. * And servers for: DNS, FTP, HTTP(+ WebSockets), Telnet. * With [SSL support](https://sming.readthedocs.io/en/latest/_inc/Sming/Components/ssl/index.html) for all network clients and servers. Based on [axTLS](http://axtls.sourceforge.net/) and [BearSSL](https://www.bearssl.org/). - * Out of the box support for OTA over HTTPS. + * Over-The-Air(OTA) firmware upgrades via HTTP(S) and MQTT(S). * ESP8266 specific features. * Integrated boot loader [rboot](https://sming.readthedocs.io/en/latest/_inc/Sming/Components/rboot/index.html) with support for 1MB ROMs, OTA firmware updating and ROM switching. * [Crash handlers](https://sming.readthedocs.io/en/latest/information/debugging.html) for analyzing/handling system restarts due to fatal errors or WDT resets. From 393ec6dd9e6d39d1e0551f39a0400a001bfb9117 Mon Sep 17 00:00:00 2001 From: Slavey Karadzhov Date: Fri, 19 Mar 2021 10:06:09 +0100 Subject: [PATCH 09/16] Applying review comments. --- Sming/Libraries/OtaUpgradeMqtt/component.mk | 8 ++-- .../OtaUpgradeMqtt/samples/Upgrade/.gitignore | 1 + .../OtaUpgradeMqtt/samples/Upgrade/README.rst | 2 +- .../samples/Upgrade/app/application.cpp | 28 +++++------- .../samples/Upgrade/basic_rboot.hw | 19 -------- .../samples/Upgrade/basic_rboot_host.hw | 19 -------- .../samples/Upgrade/component.mk | 20 +++++---- .../src/AdvancedPayloadParser.cpp | 2 +- .../OtaUpgradeMqtt/src/PayloadParser.cpp | 43 ++++++++++++------- .../OtaUpgradeMqtt/src/RbootPayloadParser.cpp | 11 +++-- .../include/OtaUpgrade/Mqtt/PayloadParser.h | 20 ++++++--- .../OtaUpgrade/Mqtt/RbootPayloadParser.h | 5 +-- 12 files changed, 82 insertions(+), 96 deletions(-) create mode 100644 Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/.gitignore delete mode 100644 Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/basic_rboot.hw delete mode 100644 Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/basic_rboot_host.hw diff --git a/Sming/Libraries/OtaUpgradeMqtt/component.mk b/Sming/Libraries/OtaUpgradeMqtt/component.mk index f7cb72545f..0bcdef89d0 100644 --- a/Sming/Libraries/OtaUpgradeMqtt/component.mk +++ b/Sming/Libraries/OtaUpgradeMqtt/component.mk @@ -2,13 +2,13 @@ COMPONENT_SRCDIRS := src/ COMPONENT_INCDIRS := src/include # If enabled (set to 1) then we can use all sofisticated mechanisms to upgrade the firmware using the ``OtaUpgrade`` library. +COMPONENT_VARS := ENABLE_OTA_ADVANCED ENABLE_OTA_ADVANCED ?= 0 -# If enabled (set to 1) then we can use unlimited number of patch versions +# If enabled (set to 1) then we can use unlimited number of patch versions +COMPONENT_VARS += ENABLE_OTA_VARINT_VERSION ENABLE_OTA_VARINT_VERSION ?= 1 ifneq ($(ENABLE_OTA_ADVANCED),0) COMPONENT_DEPENDS := OtaUpgrade -endif - -COMPONENT_VARS := ENABLE_OTA_ADVANCED ENABLE_OTA_VARINT_VERSION \ No newline at end of file +endif \ No newline at end of file diff --git a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/.gitignore b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/.gitignore new file mode 100644 index 0000000000..a662abf847 --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/.gitignore @@ -0,0 +1 @@ +ota.key diff --git a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/README.rst b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/README.rst index 85ab332ee3..ff41d9865e 100644 --- a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/README.rst +++ b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/README.rst @@ -38,7 +38,7 @@ Configuration Contains the application major and minor versions separated by comma. Example "4.2". If not set will use the current major and minor version from Sming. -.. envvar::APP_VERSION_PATCH +.. envvar:: APP_VERSION_PATCH Default: not set diff --git a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/app/application.cpp b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/app/application.cpp index a8ff6bed73..a692040eb4 100644 --- a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/app/application.cpp +++ b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/app/application.cpp @@ -1,6 +1,6 @@ #include #include -#include +#include #include #if ENABLE_OTA_ADVANCED @@ -30,11 +30,9 @@ IMPORT_FSTR(certificateData, PROJECT_DIR "/files/certificate.pem.crt.der"); Storage::Partition findRomPartition(uint8_t slot) { - String name = F("rom"); - name += slot; - auto part = Storage::findPartition(name); + auto part = Storage::spiFlash->partitions().findOta(slot); if(!part) { - debug_w("Partition '%s' not found", name.c_str()); + debug_w("Rom slot %d not found", slot); } return part; } @@ -46,11 +44,7 @@ void otaUpdate() return; } - uint8 slot; - rboot_config bootconf; - // select rom slot to flash - bootconf = rboot_get_config(); - slot = bootconf.current_rom; + uint8 slot = rboot_get_current_rom(); if(slot == 0) { slot = 1; } else { @@ -90,13 +84,13 @@ void otaUpdate() mqtt.connect(Url(MQTT_URL), "sming"); #if ENABLE_OTA_ADVANCED - /** + /* * The advanced parser suppors all firmware upgrades supported by the `OtaUpgrade` library. * It comes with firmware signing, firmware encryption and so on. */ auto parser = new OtaUpgrade::Mqtt::AdvancedPayloadParser(APP_VERSION_PATCH); #else - /** + /* * The command below uses class that stores the firmware directly * using RbootOutputStream on a location provided by us */ @@ -116,10 +110,10 @@ void otaUpdate() void showInfo() { - Serial.printf("\r\nSDK: v%s\r\n", system_get_sdk_version()); - Serial.printf("Free Heap: %d\r\n", system_get_free_heap_size()); - Serial.printf("CPU Frequency: %d MHz\r\n", system_get_cpu_freq()); - Serial.printf("System Chip ID: %x\r\n", system_get_chip_id()); + Serial.printf(_F("\r\nSDK: v%s\r\n"), system_get_sdk_version()); + Serial.printf(_F("Free Heap: %d\r\n"), system_get_free_heap_size()); + Serial.printf(_F("CPU Frequency: %d MHz\r\n"), system_get_cpu_freq()); + Serial.printf(_F("System Chip ID: %x\r\n"), system_get_chip_id()); rboot_config conf; conf = rboot_get_config(); @@ -130,7 +124,7 @@ void showInfo() debug_d("ROM 2: 0x%08x", conf.roms[2]); debug_d("GPIO ROM: %d", conf.gpio_rom); - Serial.printf("\r\nCurrently running rom %d. Application version: %s\r\n", conf.current_rom, APP_VERSION); + Serial.printf(_F("\r\nCurrently running rom %d. Application version: %s\r\n"), conf.current_rom, APP_VERSION); Serial.println(); } diff --git a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/basic_rboot.hw b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/basic_rboot.hw deleted file mode 100644 index 64465e1b3a..0000000000 --- a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/basic_rboot.hw +++ /dev/null @@ -1,19 +0,0 @@ -{ - "name": "Two ROM slots, two SPIFFS", - "base_config": "spiffs", - "partitions": { - "rom1": { - "address": "0x108000", - "size": "992K", - "type": "app", - "subtype": "ota_0", - "filename": "$(RBOOT_ROM_1_BIN)" - }, - "spiffs1": { - "address": "0x300000", - "size": "512K", - "type": "data", - "subtype": "spiffs" - } - } -} diff --git a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/basic_rboot_host.hw b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/basic_rboot_host.hw deleted file mode 100644 index 2085d420c1..0000000000 --- a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/basic_rboot_host.hw +++ /dev/null @@ -1,19 +0,0 @@ -{ - "name": "Two ROM slots, two SPIFFS", - "base_config": "spiffs", - "partitions": { - "rom1": { - "address": "0x108000", - "size": "992K", - "type": "app", - "subtype": "ota_0", - "filename": "$(BLANK_BIN)" - }, - "spiffs1": { - "address": "0x300000", - "size": "512K", - "type": "data", - "subtype": "spiffs" - } - } -} diff --git a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/component.mk b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/component.mk index 9f6d6b03da..7fe363b18f 100644 --- a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/component.mk +++ b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/component.mk @@ -11,13 +11,16 @@ APP_ID ?= "test" ## [TLS/SSL settings ] ## # Uncomment the line below to start using SSL +CONFIG_VARS := ENABLE_SSL # ENABLE_SSL := Bearssl # Set this to one if the remote firmware server requires client certificate # This option is in effect only when ENABLE_SSL is set +CONFIG_VARS += ENABLE_CLIENT_CERTIFICATE ENABLE_CLIENT_CERTIFICATE ?= 0 ## [ Firmware Update Server ] ## +CONFIG_VARS += MQTT_URL ifeq ($(MQTT_URL),) MQTT_URL := "mqtt://test.mosquitto.org:1883" ifneq ($(ENABLE_SSL),) @@ -29,24 +32,25 @@ ifeq ($(MQTT_URL),) endif endif +CONFIG_VARS += ENABLE_OTA_ADVANCED +ENABLE_OTA_ADVANCED ?= 0 ## End of user configurable settings. Don't change anything below this line -CONFIG_VARS := MQTT_URL ENABLE_SSL ENABLE_CLIENT_CERTIFICATE ENABLE_OTA_ADVANCED COMPONENT_DEPENDS := OtaUpgradeMqtt ## use rboot build mode RBOOT_ENABLED := 1 ## Use standard hardware config with two ROM slots and two SPIFFS partitions -ifeq ($(SMING_ARCH),Host) - HWCONFIG := basic_rboot_host -else - HWCONFIG := basic_rboot -endif +HWCONFIG := spiffs-two-roms -APP_CFLAGS = -DMQTT_URL="\"$(MQTT_URL)"\" -DAPP_ID="\"$(APP_ID)"\" -DENABLE_CLIENT_CERTIFICATE=$(ENABLE_CLIENT_CERTIFICATE) \ +APP_CFLAGS = -DMQTT_URL="\"$(MQTT_URL)"\" \ + -DAPP_ID="\"$(APP_ID)"\" \ + -DENABLE_CLIENT_CERTIFICATE=$(ENABLE_CLIENT_CERTIFICATE) \ -DENABLE_OTA_ADVANCED=$(ENABLE_OTA_ADVANCED) + ifneq ($(APP_VERSION),) - APP_CFLAGS += -DAPP_VERSION="\"$(APP_VERSION)"\" -DAPP_VERSION_PATCH=$(APP_VERSION_PATCH) + APP_CFLAGS += -DAPP_VERSION="\"$(APP_VERSION)"\" \ + -DAPP_VERSION_PATCH=$(APP_VERSION_PATCH) endif \ No newline at end of file diff --git a/Sming/Libraries/OtaUpgradeMqtt/src/AdvancedPayloadParser.cpp b/Sming/Libraries/OtaUpgradeMqtt/src/AdvancedPayloadParser.cpp index 6338c780b0..478b9a70d1 100644 --- a/Sming/Libraries/OtaUpgradeMqtt/src/AdvancedPayloadParser.cpp +++ b/Sming/Libraries/OtaUpgradeMqtt/src/AdvancedPayloadParser.cpp @@ -4,7 +4,7 @@ * http://github.com/SmingHub/Sming * All files of the Sming Core are provided under the LGPL v3 license. * - * AdvancedPayloadParser.h + * AdvancedPayloadParser.cpp * * Created: 2021 - Slavey Karadzhov * diff --git a/Sming/Libraries/OtaUpgradeMqtt/src/PayloadParser.cpp b/Sming/Libraries/OtaUpgradeMqtt/src/PayloadParser.cpp index 742efdad6e..7d754f65dc 100644 --- a/Sming/Libraries/OtaUpgradeMqtt/src/PayloadParser.cpp +++ b/Sming/Libraries/OtaUpgradeMqtt/src/PayloadParser.cpp @@ -1,3 +1,15 @@ +/**** + * Sming Framework Project - Open Source framework for high efficiency native ESP8266 development. + * Created 2015 by Skurydin Alexey + * http://github.com/SmingHub/Sming + * All files of the Sming Core are provided under the LGPL v3 license. + * + * PayloadParser.cpp + * + * Created: 2021 - Slavey Karadzhov + * + ****/ + #include "include/OtaUpgrade/Mqtt/PayloadParser.h" namespace OtaUpgrade @@ -6,9 +18,9 @@ namespace Mqtt { int PayloadParser::parse(MqttPayloadParserState& state, mqtt_message_t* message, const char* buffer, int length) { - if(message == nullptr) { + if(message == nullptr || buffer == nullptr) { debug_e("Invalid MQTT message"); - return 1; + return ERROR_INVALID_MQTT_MESSAGE; } if(length == MQTT_PAYLOAD_PARSER_START) { @@ -24,7 +36,7 @@ int PayloadParser::parse(MqttPayloadParserState& state, mqtt_message_t* message, auto updateState = static_cast(state.userData); if(updateState == nullptr) { debug_e("Update failed for unknown reason!"); - return -1; + return ERROR_UNKNOWN_REASON; } if(length == MQTT_PAYLOAD_PARSER_END) { @@ -49,10 +61,10 @@ int PayloadParser::parse(MqttPayloadParserState& state, mqtt_message_t* message, int patchVersion = getPatchVersion(buffer, length, offset, updateState->version); state.offset += offset; #if ENABLE_OTA_VARINT_VERSION - if(currentPatchVersion < 0) { - if(state.offset > VERSION_MAX_BYTES_ALLOWED) { + if(patchVersion < 0) { + if(state.offset > allowedVersionBytes) { debug_e("Invalid patch version."); - return -3; // + return ERROR_INVALID_PATCH_VERSION; } return 0; } @@ -75,14 +87,18 @@ int PayloadParser::parse(MqttPayloadParserState& state, mqtt_message_t* message, return 0; } - auto written = stream->write(reinterpret_cast(buffer), length); + auto written = stream->write(buffer, length); return (written - length); } -#if ENABLE_OTA_VARINT_VERSION int PayloadParser::getPatchVersion(const char* buffer, int length, size_t& offset, size_t versionStart) { + if(buffer == nullptr || length < 1) { + return ERROR_INVALID_PATCH_VERSION; + } + size_t version = versionStart; +#if ENABLE_OTA_VARINT_VERSION offset = 0; int useNextByte = 0; do { @@ -94,16 +110,13 @@ int PayloadParser::getPatchVersion(const char* buffer, int length, size_t& offse // all the data is consumed and we still don't have a version number?! return VERSION_NOT_READY; } - - return version; -} #else -int PayloadParser::getPatchVersion(const char* buffer, int length, size_t& offset, size_t versionStart) -{ offset = 1; - return buffer[0]; -} + version += buffer[0]; #endif + return version; +} + } // namespace Mqtt } // namespace OtaUpgrade diff --git a/Sming/Libraries/OtaUpgradeMqtt/src/RbootPayloadParser.cpp b/Sming/Libraries/OtaUpgradeMqtt/src/RbootPayloadParser.cpp index e2a91de07b..05ca09033d 100644 --- a/Sming/Libraries/OtaUpgradeMqtt/src/RbootPayloadParser.cpp +++ b/Sming/Libraries/OtaUpgradeMqtt/src/RbootPayloadParser.cpp @@ -4,7 +4,7 @@ * http://github.com/SmingHub/Sming * All files of the Sming Core are provided under the LGPL v3 license. * - * PayloadParser.h + * RbootPayloadParser.cpp * * Created: 2021 - Slavey Karadzhov * @@ -18,14 +18,17 @@ namespace Mqtt { bool RbootPayloadParser::switchRom(const UpdateState& updateState) { - uint8 before, after; - before = rboot_get_current_rom(); + uint8_t after; + uint8_t before = rboot_get_current_rom(); + if(before == 0) { after = 1; } else { after = 0; } - debug_d("Swapping from rom %d to rom %d.\r\n", before, after); + + debug_d("Swapping from rom %u to rom %u.\r\n", before, after); + return rboot_set_current_rom(after); } diff --git a/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/PayloadParser.h b/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/PayloadParser.h index 88ee13c952..41d63dab03 100644 --- a/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/PayloadParser.h +++ b/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/PayloadParser.h @@ -20,8 +20,11 @@ namespace OtaUpgrade { namespace Mqtt { -constexpr const uint8_t VERSION_NOT_READY = -1; -constexpr const uint8_t VERSION_MAX_BYTES_ALLOWED = 24; +constexpr int8_t VERSION_NOT_READY{-1}; + +constexpr int8_t ERROR_INVALID_MQTT_MESSAGE{-1}; +constexpr int8_t ERROR_INVALID_PATCH_VERSION{-2}; +constexpr int8_t ERROR_UNKNOWN_REASON{-10}; class PayloadParser { @@ -32,7 +35,13 @@ class PayloadParser size_t version{0}; }; - PayloadParser(size_t currentPatchVersion) : currentPatchVersion(currentPatchVersion) + /** + * @brief + * @param currentPatchVersion + * @param allowedVersionBytes - maximum allowed bytes to be used for describing a patch version + */ + PayloadParser(size_t currentPatchVersion, size_t allowedVersionBytes = 24) + : currentPatchVersion(currentPatchVersion), allowedVersionBytes(allowedVersionBytes) { } @@ -66,9 +75,10 @@ class PayloadParser int parse(MqttPayloadParserState& state, mqtt_message_t* message, const char* buffer, int length); private: - size_t currentPatchVersion; - int getPatchVersion(const char* buffer, int length, size_t& offset, size_t versionStart = 0); + + size_t currentPatchVersion; + size_t allowedVersionBytes; }; } // namespace Mqtt diff --git a/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/RbootPayloadParser.h b/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/RbootPayloadParser.h index 0289627d62..bf44c5f34b 100644 --- a/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/RbootPayloadParser.h +++ b/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/RbootPayloadParser.h @@ -4,7 +4,7 @@ * http://github.com/SmingHub/Sming * All files of the Sming Core are provided under the LGPL v3 license. * - * PayloadParser.h + * RbootPayloadParser.h * * Created: 2021 - Slavey Karadzhov * @@ -26,8 +26,7 @@ namespace Mqtt class RbootPayloadParser : public PayloadParser { public: - RbootPayloadParser(const Storage::Partition& part, size_t currentVersion) - : PayloadParser(currentVersion), part(part) + RbootPayloadParser(Storage::Partition part, size_t currentVersion) : PayloadParser(currentVersion), part(part) { } From fb215b5a7875c5e357942f816fbda5fba313d2df Mon Sep 17 00:00:00 2001 From: Slavey Karadzhov Date: Fri, 19 Mar 2021 11:50:43 +0100 Subject: [PATCH 10/16] Adding two rom mock setup for Host. --- Sming/Arch/Host/spiffs-two-roms.hw | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 Sming/Arch/Host/spiffs-two-roms.hw diff --git a/Sming/Arch/Host/spiffs-two-roms.hw b/Sming/Arch/Host/spiffs-two-roms.hw new file mode 100644 index 0000000000..80128f0191 --- /dev/null +++ b/Sming/Arch/Host/spiffs-two-roms.hw @@ -0,0 +1,13 @@ +{ + "name": "Two ROM slots with single SPIFFS", + "base_config": "spiffs", + "partitions": { + "rom1": { + "address": "0x108000", + "size": "992K", + "type": "app", + "subtype": "ota_1", + "filename": "$(BLANK_BIN)" + } + } +} From 75a68cb605d34d019693759809edf9faea0aabc0 Mon Sep 17 00:00:00 2001 From: Slavey Karadzhov Date: Fri, 19 Mar 2021 11:52:13 +0100 Subject: [PATCH 11/16] Moving fingerprint to component.mk. --- .../samples/Upgrade/app/application.cpp | 6 ++---- .../OtaUpgradeMqtt/samples/Upgrade/component.mk | 12 +++++++++--- 2 files changed, 11 insertions(+), 7 deletions(-) diff --git a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/app/application.cpp b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/app/application.cpp index a692040eb4..f02f4928cb 100644 --- a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/app/application.cpp +++ b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/app/application.cpp @@ -62,9 +62,7 @@ void otaUpdate() #ifdef ENABLE_SSL mqtt.setSslInitHandler([](Ssl::Session& session) { // These fingerprints change very frequently. - static const Ssl::Fingerprint::Cert::Sha1 sha1Fingerprint PROGMEM = {0xEE, 0xBC, 0x4B, 0xF8, 0x57, 0xE3, 0xD3, - 0xE4, 0x07, 0x54, 0x23, 0x1E, 0xF0, 0xC8, - 0xA1, 0x56, 0xE0, 0xD3, 0x1A, 0x1C}; + static const Ssl::Fingerprint::Cert::Sha1 sha1Fingerprint PROGMEM = { MQTT_FINGERPRINT_SHA1 }; // Trust certificate only if it matches the SHA1 fingerprint... session.validators.pin(sha1Fingerprint); @@ -86,7 +84,7 @@ void otaUpdate() #if ENABLE_OTA_ADVANCED /* * The advanced parser suppors all firmware upgrades supported by the `OtaUpgrade` library. - * It comes with firmware signing, firmware encryption and so on. + * `OtaUpgrade` library provides firmware signing, firmware encryption and so on. */ auto parser = new OtaUpgrade::Mqtt::AdvancedPayloadParser(APP_VERSION_PATCH); #else diff --git a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/component.mk b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/component.mk index 7fe363b18f..0d347a2601 100644 --- a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/component.mk +++ b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/component.mk @@ -32,6 +32,11 @@ ifeq ($(MQTT_URL),) endif endif +# This variable contains the SHA1 fingerprint of the SSL certificate of the MQTT server. +# It is used for certificate pinning. Make sure to change it whenever changing the MQTT_URL +CONFIG_VARS += MQTT_FINGERPRINT_SHA1 +MQTT_FINGERPRINT_SHA1 := "0xEE,0xBC,0x4B,0xF8,0x57,0xE3,0xD3,0xE4,0x07,0x54,0x23,0x1E,0xF0,0xC8,0xA1,0x56,0xE0,0xD3,0x1A,0x1C" + CONFIG_VARS += ENABLE_OTA_ADVANCED ENABLE_OTA_ADVANCED ?= 0 @@ -45,12 +50,13 @@ RBOOT_ENABLED := 1 ## Use standard hardware config with two ROM slots and two SPIFFS partitions HWCONFIG := spiffs-two-roms -APP_CFLAGS = -DMQTT_URL="\"$(MQTT_URL)"\" \ - -DAPP_ID="\"$(APP_ID)"\" \ +APP_CFLAGS = -DMQTT_URL="\"$(MQTT_URL)"\" \ + -DMQTT_FINGERPRINT_SHA1=$(MQTT_FINGERPRINT_SHA1) \ + -DAPP_ID="\"$(APP_ID)"\" \ -DENABLE_CLIENT_CERTIFICATE=$(ENABLE_CLIENT_CERTIFICATE) \ -DENABLE_OTA_ADVANCED=$(ENABLE_OTA_ADVANCED) ifneq ($(APP_VERSION),) - APP_CFLAGS += -DAPP_VERSION="\"$(APP_VERSION)"\" \ + APP_CFLAGS += -DAPP_VERSION="\"$(APP_VERSION)"\" \ -DAPP_VERSION_PATCH=$(APP_VERSION_PATCH) endif \ No newline at end of file From 39d6b1f6e23b46e9f67d81e8337e490250f3d06e Mon Sep 17 00:00:00 2001 From: Slavey Karadzhov Date: Fri, 19 Mar 2021 14:46:22 +0100 Subject: [PATCH 12/16] Don't compile AdvancedPayloadParser if not requested. --- Sming/Libraries/OtaUpgradeMqtt/component.mk | 16 +++++++++------- .../OtaUpgradeMqtt/samples/Upgrade/component.mk | 2 +- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/Sming/Libraries/OtaUpgradeMqtt/component.mk b/Sming/Libraries/OtaUpgradeMqtt/component.mk index 0bcdef89d0..546a50cf15 100644 --- a/Sming/Libraries/OtaUpgradeMqtt/component.mk +++ b/Sming/Libraries/OtaUpgradeMqtt/component.mk @@ -1,14 +1,16 @@ -COMPONENT_SRCDIRS := src/ -COMPONENT_INCDIRS := src/include +COMPONENT_SRCDIRS := +COMPONENT_SRCFILES := src/PayloadParser.cpp src/RbootPayloadParser.cpp +COMPONENT_INCDIRS := src/include # If enabled (set to 1) then we can use all sofisticated mechanisms to upgrade the firmware using the ``OtaUpgrade`` library. COMPONENT_VARS := ENABLE_OTA_ADVANCED ENABLE_OTA_ADVANCED ?= 0 +ifneq ($(ENABLE_OTA_ADVANCED),0) + COMPONENT_SRCFILES += src/AdvancedPayloadParser.cpp + COMPONENT_DEPENDS := OtaUpgrade +endif + # If enabled (set to 1) then we can use unlimited number of patch versions COMPONENT_VARS += ENABLE_OTA_VARINT_VERSION -ENABLE_OTA_VARINT_VERSION ?= 1 - -ifneq ($(ENABLE_OTA_ADVANCED),0) - COMPONENT_DEPENDS := OtaUpgrade -endif \ No newline at end of file +ENABLE_OTA_VARINT_VERSION ?= 1 \ No newline at end of file diff --git a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/component.mk b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/component.mk index 0d347a2601..806c2b46b6 100644 --- a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/component.mk +++ b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/component.mk @@ -23,7 +23,7 @@ ENABLE_CLIENT_CERTIFICATE ?= 0 CONFIG_VARS += MQTT_URL ifeq ($(MQTT_URL),) MQTT_URL := "mqtt://test.mosquitto.org:1883" - ifneq ($(ENABLE_SSL),) + ifdef ENABLE_SSL ifneq ($(ENABLE_CLIENT_CERTIFICATE),0) MQTT_URL := "mqtts://test.mosquitto.org:8884" else From a2ee07bf615cafbadc19c220e5d375d13641e6ef Mon Sep 17 00:00:00 2001 From: Slavey Karadzhov Date: Fri, 19 Mar 2021 15:27:45 +0100 Subject: [PATCH 13/16] Fixes. --- Sming/Libraries/OtaUpgradeMqtt/component.mk | 5 ++++- .../samples/Upgrade/app/application.cpp | 2 +- .../Libraries/OtaUpgradeMqtt/src/PayloadParser.cpp | 13 +++++++++---- .../include/OtaUpgrade/Mqtt/RbootPayloadParser.h | 3 ++- 4 files changed, 16 insertions(+), 7 deletions(-) diff --git a/Sming/Libraries/OtaUpgradeMqtt/component.mk b/Sming/Libraries/OtaUpgradeMqtt/component.mk index 546a50cf15..8583388890 100644 --- a/Sming/Libraries/OtaUpgradeMqtt/component.mk +++ b/Sming/Libraries/OtaUpgradeMqtt/component.mk @@ -13,4 +13,7 @@ endif # If enabled (set to 1) then we can use unlimited number of patch versions COMPONENT_VARS += ENABLE_OTA_VARINT_VERSION -ENABLE_OTA_VARINT_VERSION ?= 1 \ No newline at end of file +ENABLE_OTA_VARINT_VERSION ?= 1 + +COMPONENT_CXXFLAGS := -DENABLE_OTA_ADVANCED=$(ENABLE_OTA_ADVANCED) \ + -DENABLE_OTA_VARINT_VERSION=$(ENABLE_OTA_VARINT_VERSION) \ No newline at end of file diff --git a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/app/application.cpp b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/app/application.cpp index f02f4928cb..708f7c4c10 100644 --- a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/app/application.cpp +++ b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/app/application.cpp @@ -62,7 +62,7 @@ void otaUpdate() #ifdef ENABLE_SSL mqtt.setSslInitHandler([](Ssl::Session& session) { // These fingerprints change very frequently. - static const Ssl::Fingerprint::Cert::Sha1 sha1Fingerprint PROGMEM = { MQTT_FINGERPRINT_SHA1 }; + static const Ssl::Fingerprint::Cert::Sha1 sha1Fingerprint PROGMEM = {MQTT_FINGERPRINT_SHA1}; // Trust certificate only if it matches the SHA1 fingerprint... session.validators.pin(sha1Fingerprint); diff --git a/Sming/Libraries/OtaUpgradeMqtt/src/PayloadParser.cpp b/Sming/Libraries/OtaUpgradeMqtt/src/PayloadParser.cpp index 7d754f65dc..b63a479ea5 100644 --- a/Sming/Libraries/OtaUpgradeMqtt/src/PayloadParser.cpp +++ b/Sming/Libraries/OtaUpgradeMqtt/src/PayloadParser.cpp @@ -18,7 +18,7 @@ namespace Mqtt { int PayloadParser::parse(MqttPayloadParserState& state, mqtt_message_t* message, const char* buffer, int length) { - if(message == nullptr || buffer == nullptr) { + if(message == nullptr) { debug_e("Invalid MQTT message"); return ERROR_INVALID_MQTT_MESSAGE; } @@ -46,16 +46,21 @@ int PayloadParser::parse(MqttPayloadParserState& state, mqtt_message_t* message, delete updateState->stream; delete updateState; if(success) { - debug_d("Swtiching was successful. Restarting..."); + debug_d("Switching was successful. Restarting..."); System.restart(); } else { - debug_e("Swtiching failed!"); + debug_e("Switching failed!"); } } return 0; } + if(buffer == nullptr) { + debug_e("Invalid MQTT message"); + return ERROR_INVALID_MQTT_MESSAGE; + } + if(!updateState->started) { size_t offset = 0; int patchVersion = getPatchVersion(buffer, length, offset, updateState->version); @@ -104,7 +109,7 @@ int PayloadParser::getPatchVersion(const char* buffer, int length, size_t& offse do { version += (buffer[offset] & 0x7f); useNextByte = (buffer[offset++] & 0x80); - } while(useNextByte && (offset < length)); + } while(useNextByte && (offset < size_t(length))); if(useNextByte) { // all the data is consumed and we still don't have a version number?! diff --git a/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/RbootPayloadParser.h b/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/RbootPayloadParser.h index bf44c5f34b..68e87dd53a 100644 --- a/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/RbootPayloadParser.h +++ b/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/RbootPayloadParser.h @@ -26,7 +26,8 @@ namespace Mqtt class RbootPayloadParser : public PayloadParser { public: - RbootPayloadParser(Storage::Partition part, size_t currentVersion) : PayloadParser(currentVersion), part(part) + RbootPayloadParser(Storage::Partition part, size_t currentVersion, size_t allowedVersionBytes = 24) + : PayloadParser(currentVersion, allowedVersionBytes), part(part) { } From c7a93b22023033b2b642f39a470a25eb42da60cb Mon Sep 17 00:00:00 2001 From: Slavey Karadzhov Date: Tue, 23 Mar 2021 13:32:28 +0100 Subject: [PATCH 14/16] Added deployer tool. --- Sming/Libraries/OtaUpgradeMqtt/README.rst | 78 ++++++- Sming/Libraries/OtaUpgradeMqtt/api.rst | 1 + Sming/Libraries/OtaUpgradeMqtt/component.mk | 32 ++- .../OtaUpgradeMqtt/samples/Upgrade/README.rst | 6 + .../samples/Upgrade/app/application.cpp | 2 +- Sming/Libraries/OtaUpgradeMqtt/src/.cs | 0 .../OtaUpgradeMqtt/tools/deployer/.cproject | 97 ++++++++ .../OtaUpgradeMqtt/tools/deployer/.project | 30 +++ .../OtaUpgradeMqtt/tools/deployer/Makefile | 9 + .../OtaUpgradeMqtt/tools/deployer/README.rst | 62 +++++ .../tools/deployer/app/application.cpp | 214 ++++++++++++++++++ .../tools/deployer/component.mk | 14 ++ 12 files changed, 539 insertions(+), 6 deletions(-) delete mode 100644 Sming/Libraries/OtaUpgradeMqtt/src/.cs create mode 100644 Sming/Libraries/OtaUpgradeMqtt/tools/deployer/.cproject create mode 100644 Sming/Libraries/OtaUpgradeMqtt/tools/deployer/.project create mode 100644 Sming/Libraries/OtaUpgradeMqtt/tools/deployer/Makefile create mode 100644 Sming/Libraries/OtaUpgradeMqtt/tools/deployer/README.rst create mode 100644 Sming/Libraries/OtaUpgradeMqtt/tools/deployer/app/application.cpp create mode 100644 Sming/Libraries/OtaUpgradeMqtt/tools/deployer/component.mk diff --git a/Sming/Libraries/OtaUpgradeMqtt/README.rst b/Sming/Libraries/OtaUpgradeMqtt/README.rst index 56ff2e7808..55386d1f0a 100644 --- a/Sming/Libraries/OtaUpgradeMqtt/README.rst +++ b/Sming/Libraries/OtaUpgradeMqtt/README.rst @@ -1,7 +1,7 @@ OTA Firmware Upgrade via MQTT ============================= -.. highlight:: bash +.. highlight:: c++ Introduction ------------ @@ -9,6 +9,55 @@ Introduction This library allows Sming applications to upgrade their firmware Over-The-Air (OTA) using the MQTT protocol. MTQTT has less overhead compared to HTTP and can be used for faster delivery of application updates. +Using +----- + +1. Add ``COMPONENT_DEPENDS += OtaUpgradeMqtt`` to your application componenent.mk file. +2. Add these lines to your application:: + + #include + + #if ENABLE_OTA_ADVANCED + #include + #endif + + MqttClient mqtt; + + // Call when IP address has been obtained + void onIp(IpAddress ip, IpAddress mask, IpAddress gateway) + { + // ... + + mqtt.connect(Url(MQTT_URL), "sming"); + + #if ENABLE_OTA_ADVANCED + /* + * The advanced parser suppors all firmware upgrades supported by the `OtaUpgrade` library. + * `OtaUpgrade` library provides firmware signing, firmware encryption and so on. + */ + auto parser = new OtaUpgrade::Mqtt::AdvancedPayloadParser(APP_VERSION_PATCH); + #else + /* + * The command below uses class that stores the firmware directly + * using RbootOutputStream on a location provided by us + */ + auto parser = new OtaUpgrade::Mqtt::RbootPayloadParser(part, APP_VERSION_PATCH); + #endif + + mqtt.setPayloadParser([parser] + (MqttPayloadParserState& state, mqtt_message_t* message, const char* buffer, int length) -> int + { + return parser->parse(state, message, buffer, length); + }); + + String updateTopic = "a/test/u/4.3"; + mqtt.subscribe(updateTopic); + + // ... + } + +See the :sample:`Upgrade` sample application. + Versioning Principles --------------------- To simplify the OTA process we strongly recommend the following versioning principles for your application: @@ -29,11 +78,11 @@ Theory Of Operation Depending on the size of the new firmware and the speed of the connection an update can take 10 to 20 seconds. 2. The application connects via MQTT to a remote server and subscribes to a special topic. The topic is based on the - application id and its current version. If the current application version is 4.3.1 then the topic that will be used for OTA is ``/a/test/u/4.3``. + application id and its current version. If the current application id is ``test`` and version is ``4.3.1`` then the topic that will be used for OTA is ``a/test/u/4.3``. 3. If there is a need to support both stable and unstable/nightly builds then the topic name can have `s` or `u` suffix. For example - all stable versions should be published and downloaded from the topic ``/a/test/u/4.3/s``. For the unstable ones we can use the topic ``/a/test/u/4.3/u``. - If an application is interested in both then it can subscribe using the following pattern ``/a/test/u/4.3/+``. + all stable versions should be published and downloaded from the topic ``a/test/u/4.3/s``. For the unstable ones we can use the topic ``a/test/u/4.3/u``. + If an application is interested in both stable and unstable versions then it can subscribe using the following pattern ``a/test/u/4.3/+``. 4. The application is waiting for new firmware. When the application is on battery than it makes sense to wait for a limited time and if there is no message coming back to disconnect. @@ -51,6 +100,27 @@ One MQTT message contains: Based on the :envvar:`ENABLE_OTA_VARINT_VERSION` the patch version can be encoded either using one byte or a `varint `_. Based on :envvar:`ENABLE_OTA_ADVANCED` the firmware data can be either without any encoding or be signed and encrypted. +To simplify the packaging this library comes with a tool called ``deployer``. To create a package type the following from your application:: + + make ota-pack OTA_PATCH_VERSION=127 + +Replace 127 with the desired patch version. +If the option ``OTA_PATCH_VERSION`` is omitted from the command line then the patch version will be generated automatically and it will contain the current unix timestamp. + +Once a package is created it can be deployed to the firmware MQTT server using the command below:: + + make ota-deploy MQTT_FIRMWARE_URL=mqtt://relser:relpassword@attachix.com/a/test/u/4.3 + +The ``MQTT_FIRMWARE_URL`` above specifies that + +- protocol is: mqtt without SSL. Allowed values here are ``mqtt`` and ``mqtts``. The latter uses SSL. +- user is: relser +- password is: relpassword +- host is: attachix.com +- path is: /a/test/u/4.3. The path without leading and ending slashes is used to generate the topic name ``a/test/u/4.3``. + +Make sure to replace the MQTT_FIRMWARE_URL value with your MQTT server credentials, host and topic. + Security -------- For additional security a standard SSL/TLS can be used diff --git a/Sming/Libraries/OtaUpgradeMqtt/api.rst b/Sming/Libraries/OtaUpgradeMqtt/api.rst index 37505c0733..e9aca95651 100644 --- a/Sming/Libraries/OtaUpgradeMqtt/api.rst +++ b/Sming/Libraries/OtaUpgradeMqtt/api.rst @@ -5,3 +5,4 @@ OTA Upgrade over MQTT classes .. doxygenclass:: OtaUpgrade::Mqtt::PayloadParser .. doxygenclass:: OtaUpgrade::Mqtt::RbootPayloadParser +.. doxygenclass:: OtaUpgrade::Mqtt::AdvancedPayloadParser diff --git a/Sming/Libraries/OtaUpgradeMqtt/component.mk b/Sming/Libraries/OtaUpgradeMqtt/component.mk index 8583388890..7896ab0797 100644 --- a/Sming/Libraries/OtaUpgradeMqtt/component.mk +++ b/Sming/Libraries/OtaUpgradeMqtt/component.mk @@ -16,4 +16,34 @@ COMPONENT_VARS += ENABLE_OTA_VARINT_VERSION ENABLE_OTA_VARINT_VERSION ?= 1 COMPONENT_CXXFLAGS := -DENABLE_OTA_ADVANCED=$(ENABLE_OTA_ADVANCED) \ - -DENABLE_OTA_VARINT_VERSION=$(ENABLE_OTA_VARINT_VERSION) \ No newline at end of file + -DENABLE_OTA_VARINT_VERSION=$(ENABLE_OTA_VARINT_VERSION) + +##@Firmware Upgrade + +OTA_TOOLS := $(COMPONENT_PATH)/tools +OTA_DEPLOYMENT_TOOL = $(OTA_TOOLS)/deployer/out/Host/debug/firmware/deployer$(TOOL_EXT) + +$(OTA_DEPLOYMENT_TOOL): + $(Q) $(MAKE) -C $(OTA_TOOLS)/deployer SMING_ARCH=Host ENABLE_CUSTOM_LWIP=2 + + +# SDP = Sming Deployment Pakage +OTA_PACKAGE_EXT =.sdp + +OTA_PATCH_VERSION ?= $(shell date +%s) + +PACKAGE_IN = $(RBOOT_ROM_0_BIN) +ifneq ($(ENABLE_OTA_ADVANCED), 0) + PACKAGE_IN = $(OTA_UPGRADE_FILE) +endif +PACKAGE_OUT = $(PACKAGE_IN)$(OTA_PACKAGE_EXT) + +.PHONY: ota-pack +ota-pack: $(PACKAGE_OUT) ##Creates a deployment package from the current application (use OTA_PATCH_VERSION to specify the version number) + +$(PACKAGE_OUT): $(PACKAGE_IN) $(OTA_DEPLOYMENT_TOOL) + $(Q) $(OTA_DEPLOYMENT_TOOL) pack --debug=0 --nonet -- $(PACKAGE_IN) $(PACKAGE_OUT) $(OTA_PATCH_VERSION) $(ENABLE_OTA_VARINT_VERSION) + +.PHONY: ota-deploy +ota-deploy: $(PACKAGE_OUT) ##Uploads new firmware version of the current application (use MQTT_FIRMWARE_URL to specify the MQTT URL) + $(Q) $(OTA_DEPLOYMENT_TOOL) deploy --debug=0 -- $(PACKAGE_OUT) $(MQTT_FIRMWARE_URL) diff --git a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/README.rst b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/README.rst index ff41d9865e..cc14bab324 100644 --- a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/README.rst +++ b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/README.rst @@ -11,6 +11,12 @@ This application uses :component:`OtaUpgradeMqtt` and follows the recommended ve Based on :envvar:`ENABLE_OTA_ADVANCED` the firmware data can be either without any encoding or be signed and encrypted. +Tools +----- +There are two tools that facilitate the packiging and deployment of a new firmware. + +For more information read ``Firmware packaging`` in the documentation of the :component:`OtaUpgradeMqtt` component. + Security -------- Depending on :envvar:`ENABLE_SSL` a standard SSL/TLS can be enabled. This way diff --git a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/app/application.cpp b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/app/application.cpp index 708f7c4c10..87dbb20d31 100644 --- a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/app/application.cpp +++ b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/app/application.cpp @@ -98,7 +98,7 @@ void otaUpdate() mqtt.setPayloadParser([parser](MqttPayloadParserState& state, mqtt_message_t* message, const char* buffer, int length) -> int { return parser->parse(state, message, buffer, length); }); - String updateTopic = "/a/"; + String updateTopic = "a/"; updateTopic += APP_ID; updateTopic += "/u/"; updateTopic += APP_VERSION; diff --git a/Sming/Libraries/OtaUpgradeMqtt/src/.cs b/Sming/Libraries/OtaUpgradeMqtt/src/.cs deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/Sming/Libraries/OtaUpgradeMqtt/tools/deployer/.cproject b/Sming/Libraries/OtaUpgradeMqtt/tools/deployer/.cproject new file mode 100644 index 0000000000..fecee42f2f --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/tools/deployer/.cproject @@ -0,0 +1,97 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + make + + all + true + true + true + + + make + + rebuild + true + true + true + + + make + + flash + true + true + true + + + + diff --git a/Sming/Libraries/OtaUpgradeMqtt/tools/deployer/.project b/Sming/Libraries/OtaUpgradeMqtt/tools/deployer/.project new file mode 100644 index 0000000000..0b94d5c588 --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/tools/deployer/.project @@ -0,0 +1,30 @@ + + + FirmwareDeployer + + + SmingFramework + Libraries + Sming + + + + org.eclipse.cdt.managedbuilder.core.genmakebuilder + clean,full,incremental, + + + + + org.eclipse.cdt.managedbuilder.core.ScannerConfigBuilder + full,incremental, + + + + + + org.eclipse.cdt.core.cnature + org.eclipse.cdt.core.ccnature + org.eclipse.cdt.managedbuilder.core.managedBuildNature + org.eclipse.cdt.managedbuilder.core.ScannerConfigNature + + diff --git a/Sming/Libraries/OtaUpgradeMqtt/tools/deployer/Makefile b/Sming/Libraries/OtaUpgradeMqtt/tools/deployer/Makefile new file mode 100644 index 0000000000..ff51b6c3a7 --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/tools/deployer/Makefile @@ -0,0 +1,9 @@ +##################################################################### +#### Please don't change this file. Use component.mk instead #### +##################################################################### + +ifndef SMING_HOME +$(error SMING_HOME is not set: please configure it as an environment variable) +endif + +include $(SMING_HOME)/project.mk diff --git a/Sming/Libraries/OtaUpgradeMqtt/tools/deployer/README.rst b/Sming/Libraries/OtaUpgradeMqtt/tools/deployer/README.rst new file mode 100644 index 0000000000..97312c4c02 --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/tools/deployer/README.rst @@ -0,0 +1,62 @@ +Device_Scanner +============== + +.. highlight:: bash + +Utility to package and deploy firmware files for upgrade over MQTT +See :library:`OtaUpgradeMqtt` for details. + + +Packaging +--------- + +To scan your local network, do this:: + + make pack HOST_PARAMETERS=scan + + +This will start by scanning for all root devices ``:upnp:rootdevice``. +If you want to use a specific search type, just add it to the command line:: + + make run HOST_PARAMETERS='scan urn:schemas-upnp-org:service:ContentDirectory:1' + +This will take some time to complete. If there are any failures, you can retry +by running the command again: any descriptions which have already been fetched will +be skipped. + +You find output in the following directories: + +out/upnp/devices + Contains a hierarchical map of all found devices on your local UPnP network. + + +The schema directories are populated with modified versions of the description files. +These reflect the C++ class structures which we will later create: + +out/upnp/schema/services + All services are extracted into a directory by ``{domain}``. Each service has a separate .xml file. + + Service schema are re-usable and new ones should be added to the UPnP library so they are available for + everyone to use. + + +out/upnp/schema/devices + All devices (both root and embedded) are extracted to a separate directory ``{manufacturer}/{friendlyName}`` + with a single .xml file for each device. + + Device schema are specific to each device implementation so will generally be kept in a separate library + or application. + + +Processing existing descriptions +-------------------------------- + +You can also process device descriptions files pulled in via other means. For example: + + make run HOST_PARAMETERS='parse config/panasonic/viera dmr/ddd.xml dms/ddd.xml nrc/ddd.xml' + +The first parameter ``parse`` is the command. +The second parameter ``config/panasonic/viera`` gives the root directory for the device. +The remaining parameters are the relative locations from this directory of a device description file. +References to service files are pulled in: if they are missing, this may fail. + diff --git a/Sming/Libraries/OtaUpgradeMqtt/tools/deployer/app/application.cpp b/Sming/Libraries/OtaUpgradeMqtt/tools/deployer/app/application.cpp new file mode 100644 index 0000000000..ae8350511a --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/tools/deployer/app/application.cpp @@ -0,0 +1,214 @@ +#include +#include + +#ifdef ARCH_HOST +#ifdef __WIN32__ +#include +#endif + +#include +#include +#include +#endif + +// If you want, you can define WiFi settings globally in Eclipse Environment Variables +#ifndef WIFI_SSID +#define WIFI_SSID "PleaseEnterSSID" // Put you SSID and Password here +#define WIFI_PWD "PleaseEnterPass" +#endif + +namespace +{ +MqttClient mqtt; + +#ifdef ARCH_HOST + +size_t charsWriter(const char* buffer, size_t length) +{ + return fwrite(buffer, sizeof(char), length, stdout); +} + +template void print(const T& arg) +{ + String s(arg); + m_nputs(s.c_str(), s.length()); +} + +void println() +{ + m_puts("\r\n"); +} + +template void println(const T& arg) +{ + print(arg); + println(); +} + +int writePatchVersion(int patchVersion, bool useVarInt, ReadWriteStream* output) +{ + if(output == nullptr) { + return -1; + } + + int written = 0; + if(useVarInt) { + while(patchVersion > 0x7f) { + if(output->write(((uint8_t)(patchVersion)) | 0x80) < 0) { + return false; + } + patchVersion >>= 7; + written++; + } + if(output->write(((uint8_t)patchVersion) & 0x7f) < 0) { + return false; + } + written++; + } else { + written = output->write((uint8_t)patchVersion); + } + + return written; +} + +bool pack(const String& inputFileName, const String& outputFileName, size_t patchVersion, bool useVarInt) +{ + HostFileStream input; + input.open(inputFileName); + if(!input.fileExist()) { + m_printf(_F("ERROR: Invalid input file: %s\r\n"), inputFileName.c_str()); + return false; + } + + HostFileStream output; + output.open(outputFileName, eFO_CreateNewAlways | eFO_WriteOnly); + writePatchVersion(patchVersion, useVarInt, &output); + output.copyFrom(&input); + output.close(); + + return true; +} + +bool deploy(const String& outputFileName, const String& url) +{ + if(mqtt.isProcessing()) { + // we are still processing the data... + return false; + } + + HostFileStream* output = new HostFileStream(); + output->open(outputFileName); + if(!output->fileExist()) { + m_printf(_F("ERROR: Invalid input file: %s"), outputFileName.c_str()); + return false; + } + + WifiStation.enable(true, false); + WifiStation.config(WIFI_SSID, WIFI_PWD); + WifiAccessPoint.enable(false, false); + WifiEvents.onStationGotIP([url, output](IpAddress ip, IpAddress netmask, IpAddress gateway) { + Url mqttUrl(url); + + mqtt.connect(mqttUrl, "sming"); + mqtt.setConnectedHandler([mqttUrl, output](MqttClient& client, mqtt_message_t* message) -> int { + if(message == nullptr) { + // invalid message received + return 1; + } + + if(message->connack.return_code) { + m_printf(_F("ERROR: Connection failed. Reason: %s\r\n"), + mqtt_connect_error_string(static_cast(message->connack.return_code))); + System.restart(1000); + return 0; + } + + uint8_t retained = 1; + uint8_t QoS = 2; + uint8_t flags = (uint8_t)(retained + (QoS << 1)); + mqtt.publish(mqttUrl.Path.substring(1), output, flags); + mqtt.setPublishedHandler([](MqttClient& client, mqtt_message_t* message) -> int { + println(F("Firmware uploaded successfully.")); + System.restart(1000); + return 0; + }); + + return 0; + }); + }); + + return true; +} + +void help() +{ + println(); + println(F("Available commands:")); + println(F(" pack fileName.in fileName.out patchVersion Creates a package to be deployed on " + "firmware upgrade server.")); + println(F(" deploy fileName.out mqttUrl Deploys a deployment package to " + "firmware upgrade server.")); + println(); +} + +/* + * Return true to continue execution, false to quit. + */ +bool parseCommands() +{ + auto parameters = commandLine.getParameters(); + if(parameters.count() == 0) { + help(); + return false; + } + + String cmd = parameters[0].text; + if(cmd == "pack") { + if(parameters.count() >= 4) { + bool useVarInt = false; + if(parameters.count() > 4) { + useVarInt = strtol(parameters[4].text, nullptr, 0); + } + if(pack(parameters[1].text, parameters[2].text, strtol(parameters[3].text, nullptr, 0), useVarInt)) { + } + } + + return false; // after packaging the application can be terminated + } + + if(cmd == "deploy") { + if(parameters.count() < 2) { + m_printf(_F("ERROR: Specify package filename.\r\n")); + return false; + } + + if(parameters.count() < 3) { + m_printf(_F("ERROR: Specify MQTT_FIRMWARE_URL.\r\n")); + return false; + } + + return deploy(parameters[1].text, parameters[2].text); + } + + help(); + return false; +} + +#endif // ARCH_HOST + +} // namespace + +void init() +{ + Serial.setTxBufferSize(1024); + Serial.begin(SERIAL_BAUD_RATE); + Serial.systemDebugOutput(true); + +#ifdef ARCH_HOST + m_setPuts(charsWriter); + + if(!parseCommands()) { + System.restart(1000); + } +#endif +} diff --git a/Sming/Libraries/OtaUpgradeMqtt/tools/deployer/component.mk b/Sming/Libraries/OtaUpgradeMqtt/tools/deployer/component.mk new file mode 100644 index 0000000000..fc77dcce33 --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/tools/deployer/component.mk @@ -0,0 +1,14 @@ +COMPONENT_DEPENDS := OtaUpgradeMqtt +APP_NAME := deployer + +SMING_ARCH := Host + +##@Firmware Upgrade + +APP=$(TARGET_OUT_0) + +pack: application $(APP) ##Pack firmware files for deployment (HOST_PARAMETERS=inputFile packedFile version 1|0. Example: ) + $(Q) $(APP) pack -- $(HOST_PARAMETERS) + +deploy: application $(APP) ##Deploy firmware files to MQTT server (HOST_PARAMETERS=packedFile MQTTQ_URL. Example MQTT user: ) + $(Q) $(APP) deploy -- $(HOST_PARAMETERS) \ No newline at end of file From 3e7b001b4b447b70e9f9043427c2e40aab78590a Mon Sep 17 00:00:00 2001 From: Slavey Karadzhov Date: Fri, 26 Mar 2021 10:29:22 +0100 Subject: [PATCH 15/16] Convert always shown message to info message. --- Sming/Arch/Host/Components/hostlib/startup.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Sming/Arch/Host/Components/hostlib/startup.cpp b/Sming/Arch/Host/Components/hostlib/startup.cpp index 7b09ea3797..6349932a03 100644 --- a/Sming/Arch/Host/Components/hostlib/startup.cpp +++ b/Sming/Arch/Host/Components/hostlib/startup.cpp @@ -241,7 +241,7 @@ int main(int argc, char* argv[]) } if(config.pause != 0) { - hostmsg("If required, you may start terminal application(s) now"); + host_debug_i("If required, you may start terminal application(s) now"); pause(config.pause); } From 9e9ffe7c88c855ee3898f22e1d7db06ff309db67 Mon Sep 17 00:00:00 2001 From: Slavey Karadzhov Date: Fri, 26 Mar 2021 10:30:09 +0100 Subject: [PATCH 16/16] Documentation changes. Smuggle small change to the documentation to help Windows users install ESP32 toolchain. --- .../OtaUpgradeMqtt/tools/deployer/README.rst | 59 +------------------ docs/source/getting-started/windows/index.rst | 28 ++++++--- docs/source/index.rst | 2 +- 3 files changed, 23 insertions(+), 66 deletions(-) diff --git a/Sming/Libraries/OtaUpgradeMqtt/tools/deployer/README.rst b/Sming/Libraries/OtaUpgradeMqtt/tools/deployer/README.rst index 97312c4c02..5a0f568c0c 100644 --- a/Sming/Libraries/OtaUpgradeMqtt/tools/deployer/README.rst +++ b/Sming/Libraries/OtaUpgradeMqtt/tools/deployer/README.rst @@ -1,62 +1,7 @@ -Device_Scanner -============== +Deployer +======== .. highlight:: bash Utility to package and deploy firmware files for upgrade over MQTT See :library:`OtaUpgradeMqtt` for details. - - -Packaging ---------- - -To scan your local network, do this:: - - make pack HOST_PARAMETERS=scan - - -This will start by scanning for all root devices ``:upnp:rootdevice``. -If you want to use a specific search type, just add it to the command line:: - - make run HOST_PARAMETERS='scan urn:schemas-upnp-org:service:ContentDirectory:1' - -This will take some time to complete. If there are any failures, you can retry -by running the command again: any descriptions which have already been fetched will -be skipped. - -You find output in the following directories: - -out/upnp/devices - Contains a hierarchical map of all found devices on your local UPnP network. - - -The schema directories are populated with modified versions of the description files. -These reflect the C++ class structures which we will later create: - -out/upnp/schema/services - All services are extracted into a directory by ``{domain}``. Each service has a separate .xml file. - - Service schema are re-usable and new ones should be added to the UPnP library so they are available for - everyone to use. - - -out/upnp/schema/devices - All devices (both root and embedded) are extracted to a separate directory ``{manufacturer}/{friendlyName}`` - with a single .xml file for each device. - - Device schema are specific to each device implementation so will generally be kept in a separate library - or application. - - -Processing existing descriptions --------------------------------- - -You can also process device descriptions files pulled in via other means. For example: - - make run HOST_PARAMETERS='parse config/panasonic/viera dmr/ddd.xml dms/ddd.xml nrc/ddd.xml' - -The first parameter ``parse`` is the command. -The second parameter ``config/panasonic/viera`` gives the root directory for the device. -The remaining parameters are the relative locations from this directory of a device description file. -References to service files are pulled in: if they are missing, this may fail. - diff --git a/docs/source/getting-started/windows/index.rst b/docs/source/getting-started/windows/index.rst index d9fa84bfaa..b31bd9b026 100644 --- a/docs/source/getting-started/windows/index.rst +++ b/docs/source/getting-started/windows/index.rst @@ -23,6 +23,9 @@ Open an *administrative* **cmd.exe** command prompt and paste the text from the curl -LO https://raw.githubusercontent.com/SmingHub/Sming/develop/Tools/choco-install.cmd && choco-install.cmd +At the moment the Esp32 toolchain is not installed by default. If you want to install it run the following command:: + + choco install -y sming.esp32 .. important:: @@ -32,22 +35,31 @@ Open an *administrative* **cmd.exe** command prompt and paste the text from the It is also inadvisable to continue running with elevated privileges. +If you followed and executed carefully the steps above Sming should be installed and configured. +You can scroll down to :ref:`Build Basic_Blink` to check the installation. + +Optional step: Switch to stable version +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The installer uses the latest **develop** branch. This one contains great new features and bugfixes but can be ``unstable`` at times. + +Switching to our ``stable`` release will guarantee you that Sming's code will not change so often. +On the downside you will have to wait for all new shiny features and bugfixes. + +If you really want to use the latest ``stable`` release you can type the command below:: -The installer uses the latest **develop** branch. -The command below will get for you the latest ``stable`` release:: + cd %SMING_HOME% + git checkout master + git pull - cd %SMING_HOME% - git checkout master - git pull +Optional step: Re-installation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In case something is broken, this will perform a forced re-install of all packages:: rmdir /s /q c:\tools\sming choco install sming -y -f -x - - Packages -------- @@ -107,7 +119,7 @@ To check the installation, open a command prompt and type these commands:: make The project should build without error. - + Next steps ---------- diff --git a/docs/source/index.rst b/docs/source/index.rst index fdb883e3f1..9111286243 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -30,7 +30,7 @@ Summary - And servers for: DNS, FTP, HTTP(+ WebSockets), Telnet. - With :doc:`SSL support <_inc/Sming/Components/ssl/index>` for all network clients and servers. Based on `axTLS `__ and `BearSSL `__. - - Out of the box support for OTA over HTTPS. + - Over-The-Air(OTA) firmware upgrades via HTTP(S) and MQTT(S). - ESP8266 specific features