feat!: Rework OTA (add downgrade capability)

[Nerivec]

Features:

• Ability to downgrade as well as upgrade (using new archiving from zigbee-OTA)
• Ability to configure:
  • image block response delay (throttling during OTA updates, stable networks can reduce this, unstable ones increase it...)
  • default maximum data size (mostly to allow easier testing when an OTA fails and this is suspected as the cause)

Improvements:

• Standardize log format prefixing with IEEE and model ID
• Various code fixes

Changes:

• [BREAKING] If needed, custom CA certificates for local overrides can be passed to node via env NODE_EXTRA_CA_CERTS (like HTTPS_PROXY). Marked as breaking because technically, it is, just not sure how many users actually use this for local overrides...

Dev:

• [BREAKING] Set all configs through single function (for Z2M)
• Added jest coverage for OTA (with updated CI workflow)
• Tweak jest config to try to optimize runtime (OTA tests takes a while...)
• Remove all manuf-specific OTA classes in favor of zigbee-OTA use (allows upgrade/downgrade, no cert issue, etc.), all relevant logic merged in ota.ts
• ota property in converter is now used to either just enable OTA for a device (ota: true), or to pass specific extra metadata required for proper detection (ota: {hardwareVersionMax: 6})
  • extra metas modelId, manufacturerName, hardwareVersionMin, hardwareVersionMax can be used to override match from device as needed
  • should only be required for very specific cases, mostly for misbehaving devices/firmware files
• Improve typing
• Remove CA certs logic previously required to access some URLs for firmware download
• [BREAKING] Adjust package exports accordingly

Removable dependencies (@Koenkk I'll let you deal with this):

• axios => uses fetch built-in
• https-proxy-agent => fetch supports using the HTTPS_PROXY env directly (can't currently test it to confirm)
• tar-stream (& types) => was only needed for download from salus
• uri-js

@Koenkk run test from pre-commit is now making this a rather long process, OTA tests are nasty on length (jest just doesn't seem to handle the shear amount of calls well, probably because of all the tracking)...

CC: @jamesonuk

TODO:

• fix!: Rework OTA zigbee2mqtt#24634
• Frontend

Fix #8131
Fix #8251
Fix Koenkk/zigbee2mqtt#24464
Probably more... (misbehaving previous URLs likely caused quite a few issues that will hopefully disappear...)

[jamesonuk]

On phone so can't really see what is going on, but is it HTTP calls that are making the build / tests take a while?
Are you downloading every update every build?
Not sure if there are eTag headers from some providers or whether the servers accept If-Modified-Since headers so you only need to actually download new / changed items

[Nerivec]

All downloads are mocked to use local files (see test/stub and fetchSpy in test/ota.test.ts).

[Koenkk]

Removable dependencies (@Koenkk I'll let you deal with this):

Done!

PR LGTM!

[Nerivec]

Note for your todo list, after dropping node 18, we can drop buffer-crc32 dep too and use crc32() from zlib.

Also, ZH waitForCommand seems to have improper typing on the tsn (should be number | undefined).

[Koenkk]

Note for your todo list, after dropping node 18, we can drop buffer-crc32 dep too and use crc32() from zlib.

Supported will be dropped once it goes EOL!

Thanks!