From 508070df71f3dfb7b3339084a1e7543313a44518 Mon Sep 17 00:00:00 2001 From: Slavey Karadzhov Date: Thu, 18 Mar 2021 13:34:38 +0100 Subject: [PATCH] Moving the code to a library with some samples. --- Sming/Libraries/OtaUpgradeMqtt/.cs | 0 Sming/Libraries/OtaUpgradeMqtt/README.rst | 85 +++++++++++ Sming/Libraries/OtaUpgradeMqtt/api.rst | 7 + Sming/Libraries/OtaUpgradeMqtt/component.mk | 14 ++ .../OtaUpgradeMqtt/samples/Upgrade}/.cproject | 0 .../OtaUpgradeMqtt/samples/Upgrade/.cs | 0 .../OtaUpgradeMqtt/samples/Upgrade}/.project | 0 .../OtaUpgradeMqtt/samples/Upgrade}/Makefile | 0 .../OtaUpgradeMqtt/samples/Upgrade/README.rst | 82 ++++++++++ .../samples/Upgrade}/app/application.cpp | 143 +++--------------- .../samples/Upgrade}/basic_rboot.hw | 0 .../samples/Upgrade}/basic_rboot_host.hw | 0 .../samples/Upgrade}/component.mk | 24 ++- .../Upgrade}/files/certificate.pem.crt.der | Bin .../Upgrade}/files/private.pem.key.der | Bin Sming/Libraries/OtaUpgradeMqtt/src/.cs | 0 .../src/AdvancedPayloadParser.cpp | 44 ++++++ .../OtaUpgradeMqtt/src/PayloadParser.cpp | 109 +++++++++++++ .../OtaUpgradeMqtt/src/RbootPayloadParser.cpp | 43 ++++++ .../OtaUpgrade/Mqtt/AdvancedPayloadParser.h | 37 +++++ .../include/OtaUpgrade/Mqtt/PayloadParser.h | 75 +++++++++ .../OtaUpgrade/Mqtt/RbootPayloadParser.h | 43 ++++++ samples/Ota_Mqtt/README.rst | 105 ------------- 23 files changed, 576 insertions(+), 235 deletions(-) create mode 100644 Sming/Libraries/OtaUpgradeMqtt/.cs create mode 100644 Sming/Libraries/OtaUpgradeMqtt/README.rst create mode 100644 Sming/Libraries/OtaUpgradeMqtt/api.rst create mode 100644 Sming/Libraries/OtaUpgradeMqtt/component.mk rename {samples/Ota_Mqtt => Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade}/.cproject (100%) create mode 100644 Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/.cs rename {samples/Ota_Mqtt => Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade}/.project (100%) rename {samples/Ota_Mqtt => Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade}/Makefile (100%) create mode 100644 Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/README.rst rename {samples/Ota_Mqtt => Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade}/app/application.cpp (55%) rename {samples/Ota_Mqtt => Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade}/basic_rboot.hw (100%) rename {samples/Ota_Mqtt => Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade}/basic_rboot_host.hw (100%) rename {samples/Ota_Mqtt => Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade}/component.mk (74%) rename {samples/Ota_Mqtt => Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade}/files/certificate.pem.crt.der (100%) rename {samples/Ota_Mqtt => Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade}/files/private.pem.key.der (100%) create mode 100644 Sming/Libraries/OtaUpgradeMqtt/src/.cs create mode 100644 Sming/Libraries/OtaUpgradeMqtt/src/AdvancedPayloadParser.cpp create mode 100644 Sming/Libraries/OtaUpgradeMqtt/src/PayloadParser.cpp create mode 100644 Sming/Libraries/OtaUpgradeMqtt/src/RbootPayloadParser.cpp create mode 100644 Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/AdvancedPayloadParser.h create mode 100644 Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/PayloadParser.h create mode 100644 Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/RbootPayloadParser.h delete mode 100644 samples/Ota_Mqtt/README.rst diff --git a/Sming/Libraries/OtaUpgradeMqtt/.cs b/Sming/Libraries/OtaUpgradeMqtt/.cs new file mode 100644 index 0000000000..e69de29bb2 diff --git a/Sming/Libraries/OtaUpgradeMqtt/README.rst b/Sming/Libraries/OtaUpgradeMqtt/README.rst new file mode 100644 index 0000000000..56ff2e7808 --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/README.rst @@ -0,0 +1,85 @@ +OTA Firmware Upgrade via MQTT +============================= + +.. highlight:: bash + +Introduction +------------ + +This library allows Sming applications to upgrade their firmware Over-The-Air (OTA) using the MQTT protocol. +MTQTT has less overhead compared to HTTP and can be used for faster delivery of application updates. + +Versioning Principles +--------------------- +To simplify the OTA process we strongly recommend the following versioning principles for your application: + +1. Use `semantic versioning `_. + If your current application version is 4.3.1 then 4 is the major, 3 is the minor and 1 is the patch version number. + +2. Every application firmware knows its version. + +3. An application with the same major and minor version should be compatible for update no matter what the patch number is. + If the new firmware is not compatible then a new minor or major version should be used. + +Theory Of Operation +------------------- +1. On a period of time the application connects to check if there is a new version of the firmware. + In your application this period has to be carefully selected so that OTA updates occur when the device has + enough resources: memory, space on flash, power and time to complete such an update. Also there should be no critical task running at the moment. + Depending on the size of the new firmware and the speed of the connection an update can take 10 to 20 seconds. + +2. The application connects via MQTT to a remote server and subscribes to a special topic. The topic is based on the + application id and its current version. If the current application version is 4.3.1 then the topic that will be used for OTA is ``/a/test/u/4.3``. + +3. If there is a need to support both stable and unstable/nightly builds then the topic name can have `s` or `u` suffix. For example + all stable versions should be published and downloaded from the topic ``/a/test/u/4.3/s``. For the unstable ones we can use the topic ``/a/test/u/4.3/u``. + If an application is interested in both then it can subscribe using the following pattern ``/a/test/u/4.3/+``. + +4. The application is waiting for new firmware. When the application is on battery than it makes sense to wait for a limited time and if there is no + message coming back to disconnect. + +Firmware packaging +------------------ +The firmware update must come as one MQTT message. The MQTT protocol allows messages with a maximum size of 268435455 bytes approx 260MB. +This should be perfectly enough for a device that has maximum 1MB available for an application ROM. + +One MQTT message contains: + +- patch version of the firmware +- followed by the firmware data itself + +Based on the :envvar:`ENABLE_OTA_VARINT_VERSION` the patch version can be encoded either using one byte or a `varint `_. +Based on :envvar:`ENABLE_OTA_ADVANCED` the firmware data can be either without any encoding or be signed and encrypted. + +Security +-------- +For additional security a standard SSL/TLS can be used + +1. The communication should be secured using standard SSL. + +2. To prove that the server is the correct one: The MQTT clients should pin the public key fingerprint on the server. + OR have a list of public key fingerprints that are allowed. + +3. To prove that the clients are allowed to connect: Every MQTT client should also have a client certificate that is signed by the server. + +Configuration +------------- + +.. envvar:: ENABLE_OTA_VARINT_VERSION + + Default: 1 (enabled) + + If set to 1 the OTA upgrade mechanism and application will use a `varint `_ + encoding for the patch version. Thus allowing unlimited number of patch versions. Useful for enumerating unstable/nightly releases. + A bit more difficult to read and write but allows for unlimited versions. + + If set to 0 the OTA upgrade mechanism and application will use one byte for the patch version which will limit it to 256 possible patch versions. + Useful for enumarating stable releases. Easier to write and read but limited to 256 versions only. + +.. envvar:: ENABLE_OTA_ADVANCED + + Default: 0 (disabled) + + If set to 1 the library will work with OtaUpgradeStream which supports signature and encryption of the firmware data itself. + See :component:`OtaUpgrade` for details. In the application the AdvancedPayloadParser can be used to do the MQTT message handling. + diff --git a/Sming/Libraries/OtaUpgradeMqtt/api.rst b/Sming/Libraries/OtaUpgradeMqtt/api.rst new file mode 100644 index 0000000000..37505c0733 --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/api.rst @@ -0,0 +1,7 @@ +OTA Upgrade over MQTT classes +============================= + +.. doxygentypedef:: OtaUpgrade::Mqtt + +.. doxygenclass:: OtaUpgrade::Mqtt::PayloadParser +.. doxygenclass:: OtaUpgrade::Mqtt::RbootPayloadParser diff --git a/Sming/Libraries/OtaUpgradeMqtt/component.mk b/Sming/Libraries/OtaUpgradeMqtt/component.mk new file mode 100644 index 0000000000..f7cb72545f --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/component.mk @@ -0,0 +1,14 @@ +COMPONENT_SRCDIRS := src/ +COMPONENT_INCDIRS := src/include + +# If enabled (set to 1) then we can use all sofisticated mechanisms to upgrade the firmware using the ``OtaUpgrade`` library. +ENABLE_OTA_ADVANCED ?= 0 + +# If enabled (set to 1) then we can use unlimited number of patch versions +ENABLE_OTA_VARINT_VERSION ?= 1 + +ifneq ($(ENABLE_OTA_ADVANCED),0) + COMPONENT_DEPENDS := OtaUpgrade +endif + +COMPONENT_VARS := ENABLE_OTA_ADVANCED ENABLE_OTA_VARINT_VERSION \ No newline at end of file diff --git a/samples/Ota_Mqtt/.cproject b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/.cproject similarity index 100% rename from samples/Ota_Mqtt/.cproject rename to Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/.cproject diff --git a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/.cs b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/.cs new file mode 100644 index 0000000000..e69de29bb2 diff --git a/samples/Ota_Mqtt/.project b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/.project similarity index 100% rename from samples/Ota_Mqtt/.project rename to Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/.project diff --git a/samples/Ota_Mqtt/Makefile b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/Makefile similarity index 100% rename from samples/Ota_Mqtt/Makefile rename to Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/Makefile diff --git a/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/README.rst b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/README.rst new file mode 100644 index 0000000000..85ab332ee3 --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/README.rst @@ -0,0 +1,82 @@ +OTA over MQTT +============= + +.. highlight:: bash + +Introduction +------------ + +This example demonstrates how you can create an application that updates its firmware Over The Air (OTA) using the MQTT protocol. +This application uses :component:`OtaUpgradeMqtt` and follows the recommended versioning principles. + +Based on :envvar:`ENABLE_OTA_ADVANCED` the firmware data can be either without any encoding or be signed and encrypted. + +Security +-------- +Depending on :envvar:`ENABLE_SSL` a standard SSL/TLS can be enabled. This way + +1. The communication between the application and the server will be encrypted using standard SSL. + +2. To prove that the server is the correct one: The MQTT clients should pin the public key fingerprint on the server. + OR have a list of public key fingerprints that are allowed. + +3. Depending on :envvar:`ENABLE_CLIENT_CERTIFICATE` the application can send a client certificate that is signed by the server. + +Configuration +------------- + +.. envvar:: APP_ID + + Default: "test" + + This variable contains the unique application name. + +.. envvar:: APP_VERSION + + Default: not set + + Contains the application major and minor versions separated by comma. Example "4.2". + If not set will use the current major and minor version from Sming. + +.. envvar::APP_VERSION_PATCH + + Default: not set + + Contains the application patch version as integer. For stable versions you can use 0 until 255. + For unstable versions the current timestamp can be used as a patch version. + +.. envvar:: ENABLE_OTA_VARINT_VERSION + + Default: 1 (enabled) + + If set to 1 the OTA upgrade mechanism and application will use a `varint `_ + encoding for the patch version. Thus allowing unlimited number of patch versions. Useful for enumerating unstable/nightly releases. + A bit more difficult to read and write but allows for unlimited versions. + + If set to 0 the OTA upgrade mechanism and application will use one byte for the patch version which will limit it to 256 possible patch versions. + Useful for enumarating stable releases. Easier to write and read but limited to 256 versions only. + +.. envvar:: ENABLE_OTA_ADVANCED + + Default: 0 (disabled) + + If set to 1 the library will work with OtaUpgradeStream which supports signature and encryption of the firmware data itself. + See :component:`OtaUpgrade` for details. + +.. envvar:: ENABLE_SSL + + Default: unset (disable) + + If set to 1 (highly recommended), OTA upgrade files will be trasnferred securely over TLS/SSL. + +.. envvar:: ENABLE_CLIENT_CERTIFICATE + + Default: 0 (disabled) + + Used in combination with ``ENABLE_SSL``. Set to 1 if the remote server requires the application to authenticate via client certficate. + +.. envvar:: MQTT_URL + + Default: depends on ``ENABLE_SSL`` and ``ENABLE_CLIENT_CERTIFICATE`` values + + Url containing the location of the firmware update MQTT server. diff --git a/samples/Ota_Mqtt/app/application.cpp b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/app/application.cpp similarity index 55% rename from samples/Ota_Mqtt/app/application.cpp rename to Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/app/application.cpp index c94503a291..a8ff6bed73 100644 --- a/samples/Ota_Mqtt/app/application.cpp +++ b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/app/application.cpp @@ -1,5 +1,11 @@ #include -#include +#include +#include +#include + +#if ENABLE_OTA_ADVANCED +#include +#endif // If you want, you can define WiFi settings globally in Eclipse Environment Variables #ifndef WIFI_SSID @@ -22,16 +28,6 @@ IMPORT_FSTR(privateKeyData, PROJECT_DIR "/files/private.pem.key.der"); IMPORT_FSTR(certificateData, PROJECT_DIR "/files/certificate.pem.crt.der"); #endif -struct UpdateState { - RbootOutputStream* stream{nullptr}; - bool started{false}; - size_t offset{0}; // The bytes used for encoding the version. - size_t version{0}; -}; - -constexpr const uint8_t VERSION_NOT_READY = -1; -constexpr const uint8_t VERSION_MAX_BYTES_ALLOWED = 24; - Storage::Partition findRomPartition(uint8_t slot) { String name = F("rom"); @@ -43,47 +39,6 @@ Storage::Partition findRomPartition(uint8_t slot) return part; } -void switchRom() -{ - uint8 before, after; - before = rboot_get_current_rom(); - if(before == 0) { - after = 1; - } else { - after = 0; - } - Serial.printf("Swapping from rom %d to rom %d.\r\n", before, after); - rboot_set_current_rom(after); - Serial.println("Restarting...\r\n"); - System.restart(); -} - -#if ENABLE_VARINT_PATCH_VERSION -int getPatchVersion(const char* buffer, int length, size_t& offset, size_t versionStart = 0) -{ - size_t version = versionStart; - offset = 0; - int useNextByte = 0; - do { - version += (buffer[offset] & 0x7f); - useNextByte = (buffer[offset++] & 0x80); - } while(useNextByte && (offset < length)); - - if(useNextByte) { - // all the data is consumed and we still don't have a version number?! - return VERSION_NOT_READY; - } - - return version; -} -#else -int getPatchVersion(const char* buffer, int length, size_t& offset, size_t versionStart = 0) -{ - offset = 1; - return buffer[0]; -} -#endif - void otaUpdate() { if(mqtt.isProcessing()) { @@ -133,79 +88,23 @@ void otaUpdate() #endif mqtt.connect(Url(MQTT_URL), "sming"); - mqtt.setPayloadParser( - [part](MqttPayloadParserState& state, mqtt_message_t* message, const char* buffer, int length) -> int { - if(message == nullptr) { - debug_e("Invalid MQTT message"); - return 1; - } - - if(length == MQTT_PAYLOAD_PARSER_START) { - UpdateState* updateState = new UpdateState(); - updateState->stream = nullptr; - updateState->started = false; - - state.offset = 0; - state.userData = updateState; - return 0; - } - - auto updateState = static_cast(state.userData); - if(updateState == nullptr) { - debug_e("Update failed for unknown reason!"); - return -1; - } - - if(length == MQTT_PAYLOAD_PARSER_END) { - bool skip = (updateState->stream == nullptr); - if(!skip) { - delete updateState->stream; - switchRom(); - } - - return 0; - } - if(!updateState->started) { - size_t offset = 0; - int patchVersion = getPatchVersion(buffer, length, offset, updateState->version); - updateState->offset += offset; -#if ENABLE_VARINT_PATCH_VERSION - if(patchVersion == VERSION_NOT_READY) { - - if(updateState->offset > VERSION_MAX_BYTES_ALLOWED) { - debug_e("Invalid patch version."); - return -3; // - } - return 0; - } +#if ENABLE_OTA_ADVANCED + /** + * The advanced parser suppors all firmware upgrades supported by the `OtaUpgrade` library. + * It comes with firmware signing, firmware encryption and so on. + */ + auto parser = new OtaUpgrade::Mqtt::AdvancedPayloadParser(APP_VERSION_PATCH); +#else + /** + * The command below uses class that stores the firmware directly + * using RbootOutputStream on a location provided by us + */ + auto parser = new OtaUpgrade::Mqtt::RbootPayloadParser(part, APP_VERSION_PATCH); #endif - updateState->started = true; - if(patchVersion < APP_VERSION_PATCH) { - // The update is not newer than our current patch version - return 0; - } - - if(message->common.length - updateState->offset > part.size()) { - debug_e("The new rom is too big to fit!"); - return -2; - } - - length -= offset; - buffer += offset; - - updateState->stream = new RbootOutputStream(part.address(), part.size()); - } - - auto rbootStream = static_cast(updateState->stream); - if(rbootStream == nullptr) { - return 0; - } - - auto written = rbootStream->write(reinterpret_cast(buffer), length); - return (written - length); - }); + mqtt.setPayloadParser([parser](MqttPayloadParserState& state, mqtt_message_t* message, const char* buffer, + int length) -> int { return parser->parse(state, message, buffer, length); }); String updateTopic = "/a/"; updateTopic += APP_ID; diff --git a/samples/Ota_Mqtt/basic_rboot.hw b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/basic_rboot.hw similarity index 100% rename from samples/Ota_Mqtt/basic_rboot.hw rename to Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/basic_rboot.hw diff --git a/samples/Ota_Mqtt/basic_rboot_host.hw b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/basic_rboot_host.hw similarity index 100% rename from samples/Ota_Mqtt/basic_rboot_host.hw rename to Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/basic_rboot_host.hw diff --git a/samples/Ota_Mqtt/component.mk b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/component.mk similarity index 74% rename from samples/Ota_Mqtt/component.mk rename to Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/component.mk index 2c97a0be95..9f6d6b03da 100644 --- a/samples/Ota_Mqtt/component.mk +++ b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/component.mk @@ -1,10 +1,6 @@ ## User configurable settings -ENABLE_CLIENT_CERTIFICATE ?= 0 -ENABLE_VARINT_PATCH_VERSION ?=0 - -CONFIG_VARS := MQTT_URL ENABLE_SSL ENABLE_CLIENT_CERTIFICATE ENABLE_VARINT_PATCH_VERSION - +## [Application id and version] ## # Application id APP_ID ?= "test" @@ -13,7 +9,15 @@ APP_ID ?= "test" # Application patch version: integer containing only the patch version # APP_VERSION_PATCH := 3 -# Firmware Update Server +## [TLS/SSL settings ] ## +# Uncomment the line below to start using SSL +# ENABLE_SSL := Bearssl + +# Set this to one if the remote firmware server requires client certificate +# This option is in effect only when ENABLE_SSL is set +ENABLE_CLIENT_CERTIFICATE ?= 0 + +## [ Firmware Update Server ] ## ifeq ($(MQTT_URL),) MQTT_URL := "mqtt://test.mosquitto.org:1883" ifneq ($(ENABLE_SSL),) @@ -25,8 +29,12 @@ ifeq ($(MQTT_URL),) endif endif + ## End of user configurable settings. Don't change anything below this line +CONFIG_VARS := MQTT_URL ENABLE_SSL ENABLE_CLIENT_CERTIFICATE ENABLE_OTA_ADVANCED +COMPONENT_DEPENDS := OtaUpgradeMqtt + ## use rboot build mode RBOOT_ENABLED := 1 @@ -38,7 +46,7 @@ else endif APP_CFLAGS = -DMQTT_URL="\"$(MQTT_URL)"\" -DAPP_ID="\"$(APP_ID)"\" -DENABLE_CLIENT_CERTIFICATE=$(ENABLE_CLIENT_CERTIFICATE) \ - -DENABLE_VARINT_PATCH_VERSION=$(ENABLE_VARINT_PATCH_VERSION) + -DENABLE_OTA_ADVANCED=$(ENABLE_OTA_ADVANCED) ifneq ($(APP_VERSION),) APP_CFLAGS += -DAPP_VERSION="\"$(APP_VERSION)"\" -DAPP_VERSION_PATCH=$(APP_VERSION_PATCH) -endif \ No newline at end of file +endif \ No newline at end of file diff --git a/samples/Ota_Mqtt/files/certificate.pem.crt.der b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/files/certificate.pem.crt.der similarity index 100% rename from samples/Ota_Mqtt/files/certificate.pem.crt.der rename to Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/files/certificate.pem.crt.der diff --git a/samples/Ota_Mqtt/files/private.pem.key.der b/Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/files/private.pem.key.der similarity index 100% rename from samples/Ota_Mqtt/files/private.pem.key.der rename to Sming/Libraries/OtaUpgradeMqtt/samples/Upgrade/files/private.pem.key.der diff --git a/Sming/Libraries/OtaUpgradeMqtt/src/.cs b/Sming/Libraries/OtaUpgradeMqtt/src/.cs new file mode 100644 index 0000000000..e69de29bb2 diff --git a/Sming/Libraries/OtaUpgradeMqtt/src/AdvancedPayloadParser.cpp b/Sming/Libraries/OtaUpgradeMqtt/src/AdvancedPayloadParser.cpp new file mode 100644 index 0000000000..6338c780b0 --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/src/AdvancedPayloadParser.cpp @@ -0,0 +1,44 @@ +/**** + * Sming Framework Project - Open Source framework for high efficiency native ESP8266 development. + * Created 2015 by Skurydin Alexey + * http://github.com/SmingHub/Sming + * All files of the Sming Core are provided under the LGPL v3 license. + * + * AdvancedPayloadParser.h + * + * Created: 2021 - Slavey Karadzhov + * + ****/ + +#include "include/OtaUpgrade/Mqtt/AdvancedPayloadParser.h" + +namespace OtaUpgrade +{ +namespace Mqtt +{ +bool AdvancedPayloadParser::switchRom(const UpdateState& updateState) +{ + if(updateState.stream == nullptr) { + return false; + } + + auto otaStream = static_cast(updateState.stream); + if(otaStream == nullptr) { + return false; + } + + if(otaStream->hasError()) { + debug_e("Got error: %s", toString(otaStream->errorCode).c_str()); + return false; + } + + return true; +} + +ReadWriteStream* AdvancedPayloadParser::getStorageStream(size_t storageSize) +{ + return new OtaUpgradeStream(); +} + +} // namespace Mqtt +} // namespace OtaUpgrade diff --git a/Sming/Libraries/OtaUpgradeMqtt/src/PayloadParser.cpp b/Sming/Libraries/OtaUpgradeMqtt/src/PayloadParser.cpp new file mode 100644 index 0000000000..742efdad6e --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/src/PayloadParser.cpp @@ -0,0 +1,109 @@ +#include "include/OtaUpgrade/Mqtt/PayloadParser.h" + +namespace OtaUpgrade +{ +namespace Mqtt +{ +int PayloadParser::parse(MqttPayloadParserState& state, mqtt_message_t* message, const char* buffer, int length) +{ + if(message == nullptr) { + debug_e("Invalid MQTT message"); + return 1; + } + + if(length == MQTT_PAYLOAD_PARSER_START) { + UpdateState* updateState = new UpdateState(); + updateState->stream = nullptr; + updateState->started = false; + + state.offset = 0; + state.userData = updateState; + return 0; + } + + auto updateState = static_cast(state.userData); + if(updateState == nullptr) { + debug_e("Update failed for unknown reason!"); + return -1; + } + + if(length == MQTT_PAYLOAD_PARSER_END) { + bool skip = (updateState->stream == nullptr); + if(!skip) { + bool success = switchRom(*updateState); + delete updateState->stream; + delete updateState; + if(success) { + debug_d("Swtiching was successful. Restarting..."); + System.restart(); + } else { + debug_e("Swtiching failed!"); + } + } + + return 0; + } + + if(!updateState->started) { + size_t offset = 0; + int patchVersion = getPatchVersion(buffer, length, offset, updateState->version); + state.offset += offset; +#if ENABLE_OTA_VARINT_VERSION + if(currentPatchVersion < 0) { + if(state.offset > VERSION_MAX_BYTES_ALLOWED) { + debug_e("Invalid patch version."); + return -3; // + } + return 0; + } +#endif + + updateState->started = true; + if(size_t(patchVersion) < currentPatchVersion) { + // The update is not newer than our current patch version + return 0; + } + + length -= offset; + buffer += offset; + + updateState->stream = getStorageStream(message->common.length - offset); + } + + auto stream = updateState->stream; + if(stream == nullptr) { + return 0; + } + + auto written = stream->write(reinterpret_cast(buffer), length); + return (written - length); +} + +#if ENABLE_OTA_VARINT_VERSION +int PayloadParser::getPatchVersion(const char* buffer, int length, size_t& offset, size_t versionStart) +{ + size_t version = versionStart; + offset = 0; + int useNextByte = 0; + do { + version += (buffer[offset] & 0x7f); + useNextByte = (buffer[offset++] & 0x80); + } while(useNextByte && (offset < length)); + + if(useNextByte) { + // all the data is consumed and we still don't have a version number?! + return VERSION_NOT_READY; + } + + return version; +} +#else +int PayloadParser::getPatchVersion(const char* buffer, int length, size_t& offset, size_t versionStart) +{ + offset = 1; + return buffer[0]; +} +#endif + +} // namespace Mqtt +} // namespace OtaUpgrade diff --git a/Sming/Libraries/OtaUpgradeMqtt/src/RbootPayloadParser.cpp b/Sming/Libraries/OtaUpgradeMqtt/src/RbootPayloadParser.cpp new file mode 100644 index 0000000000..e2a91de07b --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/src/RbootPayloadParser.cpp @@ -0,0 +1,43 @@ +/**** + * Sming Framework Project - Open Source framework for high efficiency native ESP8266 development. + * Created 2015 by Skurydin Alexey + * http://github.com/SmingHub/Sming + * All files of the Sming Core are provided under the LGPL v3 license. + * + * PayloadParser.h + * + * Created: 2021 - Slavey Karadzhov + * + ****/ + +#include "include/OtaUpgrade/Mqtt/RbootPayloadParser.h" + +namespace OtaUpgrade +{ +namespace Mqtt +{ +bool RbootPayloadParser::switchRom(const UpdateState& updateState) +{ + uint8 before, after; + before = rboot_get_current_rom(); + if(before == 0) { + after = 1; + } else { + after = 0; + } + debug_d("Swapping from rom %d to rom %d.\r\n", before, after); + return rboot_set_current_rom(after); +} + +ReadWriteStream* RbootPayloadParser::getStorageStream(size_t storageSize) +{ + if(storageSize > part.size()) { + debug_e("The new rom is too big to fit!"); + return nullptr; + } + + return new RbootOutputStream(part.address(), part.size()); +} + +} // namespace Mqtt +} // namespace OtaUpgrade diff --git a/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/AdvancedPayloadParser.h b/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/AdvancedPayloadParser.h new file mode 100644 index 0000000000..1cf01191cc --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/AdvancedPayloadParser.h @@ -0,0 +1,37 @@ +/**** + * Sming Framework Project - Open Source framework for high efficiency native ESP8266 development. + * Created 2015 by Skurydin Alexey + * http://github.com/SmingHub/Sming + * All files of the Sming Core are provided under the LGPL v3 license. + * + * AdvancedPayloadParser.h + * + * Created: 2021 - Slavey Karadzhov + * + ****/ + +#pragma once + +#include "PayloadParser.h" +#include + +namespace OtaUpgrade +{ +namespace Mqtt +{ +/** + * This parser allows the processing of firmware data + * that can be encrypted or have signature + */ +class AdvancedPayloadParser : public PayloadParser +{ +public: + using PayloadParser::PayloadParser; + + bool switchRom(const UpdateState& updateState) override; + + ReadWriteStream* getStorageStream(size_t storageSize) override; +}; + +} // namespace Mqtt +} // namespace OtaUpgrade diff --git a/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/PayloadParser.h b/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/PayloadParser.h new file mode 100644 index 0000000000..88ee13c952 --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/PayloadParser.h @@ -0,0 +1,75 @@ +/**** + * Sming Framework Project - Open Source framework for high efficiency native ESP8266 development. + * Created 2015 by Skurydin Alexey + * http://github.com/SmingHub/Sming + * All files of the Sming Core are provided under the LGPL v3 license. + * + * PayloadParser.h + * + * Created: 2021 - Slavey Karadzhov + * + ****/ + +#pragma once + +#include +#include +#include + +namespace OtaUpgrade +{ +namespace Mqtt +{ +constexpr const uint8_t VERSION_NOT_READY = -1; +constexpr const uint8_t VERSION_MAX_BYTES_ALLOWED = 24; + +class PayloadParser +{ +public: + struct UpdateState { + ReadWriteStream* stream{nullptr}; + bool started{false}; + size_t version{0}; + }; + + PayloadParser(size_t currentPatchVersion) : currentPatchVersion(currentPatchVersion) + { + } + + virtual ~PayloadParser() + { + } + + /** + * @brief This method is responsible for switching the rom. + * This method is NOT restarting the system. It will happen lated in the parse method + * @retval true if the switch was successful + */ + virtual bool switchRom(const UpdateState& updateState) = 0; + + /** + * @brief Creates new stream to store the firmware update + * @param size_t storageSize the requested storage size + * + * @retval ReadWriteStream* + */ + virtual ReadWriteStream* getStorageStream(size_t storageSize) = 0; + + /** + * @brief This method takes care to read the incoming MQTT message and pass it to the stream that + * is responsoble for storing the data. + * + * @retval int + * 0 when everything is ok + * <0 when an error has occurred + */ + int parse(MqttPayloadParserState& state, mqtt_message_t* message, const char* buffer, int length); + +private: + size_t currentPatchVersion; + + int getPatchVersion(const char* buffer, int length, size_t& offset, size_t versionStart = 0); +}; + +} // namespace Mqtt +} // namespace OtaUpgrade diff --git a/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/RbootPayloadParser.h b/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/RbootPayloadParser.h new file mode 100644 index 0000000000..0289627d62 --- /dev/null +++ b/Sming/Libraries/OtaUpgradeMqtt/src/include/OtaUpgrade/Mqtt/RbootPayloadParser.h @@ -0,0 +1,43 @@ +/**** + * Sming Framework Project - Open Source framework for high efficiency native ESP8266 development. + * Created 2015 by Skurydin Alexey + * http://github.com/SmingHub/Sming + * All files of the Sming Core are provided under the LGPL v3 license. + * + * PayloadParser.h + * + * Created: 2021 - Slavey Karadzhov + * + ****/ + +#pragma once + +#include "PayloadParser.h" +#include + +namespace OtaUpgrade +{ +namespace Mqtt +{ +/** + * @brief This parser allows the processing of firmware data that is directly stored + * to the flash memory using RbootOutputStream. + */ +class RbootPayloadParser : public PayloadParser +{ +public: + RbootPayloadParser(const Storage::Partition& part, size_t currentVersion) + : PayloadParser(currentVersion), part(part) + { + } + + bool switchRom(const UpdateState& updateState) override; + + ReadWriteStream* getStorageStream(size_t storageSize) override; + +private: + Storage::Partition part; +}; + +} // namespace Mqtt +} // namespace OtaUpgrade diff --git a/samples/Ota_Mqtt/README.rst b/samples/Ota_Mqtt/README.rst deleted file mode 100644 index ec2aa86109..0000000000 --- a/samples/Ota_Mqtt/README.rst +++ /dev/null @@ -1,105 +0,0 @@ -OTA over MQTT -============= - -.. highlight:: bash - -Introduction ------------- - -This example demonstrates how you can create an application that updates its firmware Over The Air (OTA) using the MQTT protocol. -MTQTT has less overhead compared to HTTP and can be used for faster delivery of application updates. - -Versioning ----------- -To simplify the OTA process and make it less error prone we are using the following versioning principles: -1) For version naming we use `semantic versioning `_. -If our current version is 4.3.1 then 4 is the major, 3 is the minor and 1 is the patch version number. -2) Every application firmware knows its version. -3) An application with the same major and minor version should be compatible for update no matter what the patch number is. -If the new firmware is not compatible then a new minor or major version should be used. - -Theory Of Operation -------------------- -1) On a period of time the application connects to check if there is a new version of the firmware. -In your application this period has to be carefully selected so that OTA updates occur when the device has -enough resources: memory, space on flash, power and time to complete such an update. Also there should be no critical task running at the moment. -Depending on the size of the new firmware and the speed of the connection an update can take 10 to 20 seconds. -2) The application connects via MQTT to a remote server and subscribes to a special topic. The topic is based on the -application id and its current version. If the current application version is 4.3.1 then the topic that will be used for OTA is "/a/test/u/4.3". -2.1) If there is a need to support both stable and unstable/nightly builds then the topic name can have s or u suffix. For example -all stable versions should be published and downloaded from the topic "/a/test/u/4.3/s". For the unstable ones we can use the topic "/a/test/u/4.3/u". -If an application is interested in both then it can subscribe using the following pattern "/a/test/u/4.3/+". -3) The application is waiting for new firmware. When the application is on battery than it makes sense to wait for a limited time and if there is no -message coming back to disconnect. - -Firmware packaging ------------------- -The firmware update must come as one MQTT message. The MQTT protocol allows messages with a maximum size of 268435455 bytes approx 260MB. -This should be perfectly enough for a device that has maximum 1MB available for an application ROM. -The message coming from MQTT contains: -- at the start one byte with the patch version of the firmware -- followed by the firmware data itself -For simplicity the patch version is just one byte. This limits us to 256 possible patch versions. -If needed the patch version can be encoded using `varint `_. -In this example there is no encoding, checksum, signature or encryption for the firmware data itself. - -Security --------- -For additional security a standard SSL/TLS can be used -1) The communication should be secured using standard SSL. -2) To prove that the server is the correct one: The MQTT clients should pin the public key fingerprint on the server. -OR have a list of public key fingerprints that are allowed. -3) To prove that the clients are allowed to connect: Every MQTT client should also have a client certificate that is signed by the server. - - -Configuration and Security features ------------------------------------ - -.. envvar:: APP_ID - - Default: "test" - - This variable contains the unique application name. - -.. envvar:: APP_VERSION - - Default: not set - - Contains the application major and minor versions separated by comma. Example "4.2". - If not set will use the current major and minor version from Sming. - -.. envvar::APP_VERSION_PATCH - - Default: not set - - Contains the application patch version as integer. For stable versions you can use 0 until 255. - For unstable versions the current timestamp can be used as a patch version. - -.. envvar:: ENABLE_VARINT_PATCH_VERSION - - Default: 0 (disabled) - - If set to 1 the OTA upgrade mechanism and application will use a `varint `_` - encoding for the patch version. Thus allowing unlimited number of patch versions. Useful for enumerating unstable/nightly releases. - A bit more difficult to read and write but allows for unlimited versions. - - If set to 0 the OTA upgrade mechanism and application will use one byte for the patch version which will limit it to 256 possible patch versions. - Useful for enumarating stable releases. Easier to write and read but limited to 256 versions only. - -.. envvar:: ENABLE_SSL - - Default: unset (disable) - - If set to 1 (highly recommended), OTA upgrade files will be trasnferred securely over TLS/SSL. - -.. envvar:: ENABLE_CLIENT_CERTIFICATE - - Default: 0 (disabled) - - Used in combination with ``ENABLE_SSL``. Set to 1 if the remote server requires the application to authenticate via client certficate. - -.. envvar:: MQTT_URL - - Default: depends on ``ENABLE_SSL`` and ``ENABLE_CLIENT_CERTIFICATE`` values - - Url containing the location of the firmware update MQTT server.