Sonoff Dongle P with Zigbee2mqtt can’t be swapped / replaced

[dafunkydan]

I found out i can’t just swap my Sticks (both Sonoff Zigbee Dongle Plus P) and control my Network.
After stopping Z2M, i'm waiting a couple of Minutes, plugging in the new Stick, but none of my Devices is controllable with it.
I think the relevant Part in the Z2M Logs is:
(Data request failed with error: ‘No network route’ (205)))

As soon as i plug the old one in, everything is (almost instantly) back to normal.

• The Coordinators IEEE-Adress of the new Stick is set to the one of the first Stick
• Both are on the same Firmware as well
• Since i try everthing on the same System, of course everything with Z2M is the same as well

My Goal is to have a Second spare Zigbee Dongle, ready to take over with a second spare RPi with HAOS, in Case of a Failure.
On my old Conbee II/Deconz System, i just had to make sure they are not running at the same Time, and sometimes had to wait a couple of Minutes til all Devices were controllable again with the new Stick.

What could i be looking after, did i miss an important Step?

[tommyjlong]

I am interested in this scenario too, but have a couple of questions....
As I understand it, z2m restores the "replacement"/'new" Adapter using the configuration_backup.json.

1. When does this restore actually take place as I can find no z2m "command" to do a restore? Done on Z2M startup?
2. IEEE Address can exist in TI Adapters in 3 different locations (from what I've researched):
   i) Primary IEEEE Address which is permanent on the device,
   ii) Secondary IEEE Address which is stored in a programmable part of the device. (Dicussed in the docs)
   iii) NVRAM is another place the IEEE address can be stored
   And the preferred choice is in reverse order (NVRAM IEEE Address picked over Secondary)
   When z2m restores the coordinator's information, does the IEEE Address in the configuration_backup.json file get stored in NVRAM?

[dafunkydan]

That's interesting, neither heard about that NVRAM, nor the preferring Order of it!
Maybe this is the Culprit? I just don't know what i would have to / could change, in Order to just take over with Stick 2.
At least i can say this is definitly possible with Zigbee, as i did that all the Time with my Conbee II

Regarding 1. - if i understand https://www.zigbee2mqtt.io/guide/faq/#how-do-i-migrate-from-one-adapter-to-another it should get restored on Z2M. It says:

f re-pairing is required: remove data/coordinator_backup.json (if it exists) and data/database.db
If re-pairing is not required: copy the ieee address of the old adapter into the new one

I translate it for myself, that if you would need to re-pair your Stick you would need to remove backup / db, as it would get restored after starting z2m.

[tommyjlong]

Here is my write-up on getting a "Replacement" Sonoff Dongle -P working as a replacement for an "Operational" Sonoff Dongle -P.
Note: "Data" folder in Home Assistant is located at <ha-core-config-dir>/zigbee2mqtt

1. Do a "Data" folder backup. Z2M's UI->Settings->Tools->Request Z2M backup ..wait a while until you see the button change to "download_z2m_backup". Click on it and save the zip file somewhere. The Request also updates the coordinator_backup.json file which should be used in the "restore" process for the new "Replacement" Adapter.
2. Get the "Operational" Dongle's IEEE Address
   i) Goto the "Data" folder (as suggested in the docs) and look inside the database.db file looking for :"type":"Coordinator","ieeeAddr":"0x00124b002a9a8637".
   ii) Look inside the coordinator_backup.json file and confirm it has the same IEEE address: "coordinator_ieee": "00124b002a9a8637"
3. Get the flash file used to flash the "Operational" Adapter. If you are using the firmware that came with the "Operational" Adapter, get the "Operational" Adapter's Firmware version. I look at the z2m logs (in this example its 20221226).
   Also, note the first line stating "resumed" (more on this later).

info  2024-01-05 15:06:48: zigbee-herdsman started (resumed)
info  2024-01-05 15:06:48: Coordinator firmware version: '{"meta":{"maintrel":1,"majorrel":2,"minorrel":7,"product":1,"revision":20221226,"transportrev":2},"type":"zStack3x0"}'

5. Flash the "Replacement" Adapter and set its IEEE Address. Here I use cc2538-bsl. Note that the format of the IEEE address for the cc2538-bsl uses ":". I'm using the same firmware file to flash the "Replacement" Adapter, that I used to flash the "Operational" Adapter.

$python3 cc2538-bsl.py --bootloader-sonoff-usb -e -w -v --ieee-address 00:12:4b:00:2a:9a:86:37 CC1352P2_CC2652P_launchpad_coordinator_20221226.hex

sonoff
Opening port /dev/ttyUSB0, baud 500000
Reading data from CC1352P2_CC2652P_launchpad_coordinator_20221226.hex
Firmware file: Intel Hex
Connecting to target...
CC1350 PG2.0 (7x7mm): 352KB Flash, 20KB SRAM, CCFG.BL_CONFIG at 0x00057FD8
Primary IEEE Address: 00:12:4B:00:2A:2E:2D:F5
    Performing mass erase
Erasing all main bank flash sectors
    Erase done
Writing 360448 bytes starting at address 0x00000000
Write 104 bytes at 0x00057F980
    Write done                                
Verifying by comparing CRC32 calculations.
    Verified (match: 0xa9dc145d)
Setting IEEE address to 00:12:4b:00:2a:9a:86:37
Writing 8 bytes starting at address 0x00057FC8
Write 8 bytes at 0x00057FC8
    Set address done

Note the line: Setting IEEE address to 00:12:4b:00:2a:9a:86:37. For these Texas Instrument based Adapters, this is setting what is called the "Secondary" IEEE address which cc2538-bsl sets in a "configuration" section of Flash memory and it takes precedence over the Primary IEEE Address which is a permanent address of the "Replacement" Adapter and thus can't be changed.
6. Stop Z2M. Make note of the current file path for the "Operational" Adapter.
7. Plug-in the "Replacement" Adapter and note its new file path and configure Z2M to use this new file path.
8. Startup Z2M. Note in the log's startup that the adapter firmware is the same as the "Operational" Adapter and that the Adapter is being "restored". From what I can tell, at startup, Z2M is looking to see if the Adapter has any information in the Adapter's "Network Information Base" (network keys, devices, etc.) and if not, it will perform a "restore" procedure from the coordinator_backup.json file.

info  2024-01-15 13:27:21: zigbee-herdsman started (restored)
info  2024-01-15 13:27:21: Coordinator firmware version: '{"meta":{"maintrel":1,"majorrel":2,"minorrel":7,"product":1,"revision":20221226,"transportrev":2},"type":"zStack3x0"}'

9. From here I started turning on/off devices starting with the ones nearest the Coordinator, and worked towards devices further away.

[dafunkydan]

Wow, Dude, that is incredible. Awesome. Too bad i can only mark it with only one 🚀 ❤️ 🎉.
Thank you so much, not only for digging into that Thing, but also taking your Time to precisly document and share this. Highly appreciated!!! ❤️
Unfortunatly i can follow your Steps (and of Course give Feedback) next Weekend earliest - i both need some Focus Time, and my Girlfriend not beeing Home 😆 Just be sure you just earned a big bag of Karmapoints. Thanks Mate!!

[comk22]

If you have the time please add a tutorial post on the Hassio board this is great advice and not easy to find. I'm planning to make a backup/failover setup with a second identical dongle and this helped me a lot. After bricking one dongle (because I was not paying attention) I was able to make the 2nd one work. Thanks a lot!

[tommyjlong]

Here is another reference also on this topic....
Koenkk/Z-Stack-firmware#72 (comment)

[Hedda]

For reference, this was also cross-posted and discussed in Home Assistant's community forum here -> https://community.home-assistant.io/t/sonoff-dongle-p-with-zigbee2mqtt-cant-be-replaced/669878/14

• Do a "Data" folder backup. Z2M's UI->Settings->Tools->Request Z2M backup ..wait a while until you see the button change to "download_z2m_backup". Click on it and save the zip file somewhere. The Request also updates the coordinator_backup.json file which should be used in the "restore" process for the new "Replacement" Adapter.
• Get the "Operational" Dongle's IEEE Address
  i) Goto the "Data" folder (as suggested in the docs) and look inside the database.db file looking for :"type":"Coordinator","ieeeAddr":"0x00124b002a9a8637".
  ii) Look inside the coordinator_backup.json file and confirm it has the same IEEE address: "coordinator_ieee": "00124b002a9a8637"

FYI, you can alternatively restore coordinator_backup.json file using zigpy-cli to write it to new Zigbee Coordinator radio apdapter type.

See -> #16478

That method is manual + unofficial but the benefit is can restore ít to any adapter type so not limited to restoring from or to the same type.

Note! Both zigpy library and zigbee-herdsman library that Zigbee2MQTT depends on use "Open ZigBee Coordinator Backup Format":

https://github.com/zigpy/open-coordinator-backup

zigpy-cli can be used as a stand-alone tool for "Open ZigBee Coordinator Backup Format" to backup/restore Zigbee network from NVRAM:

https://github.com/zigpy/zigpy-cli

https://github.com/zigpy/zigpy-cli/blob/dev/README.md#network-commands

Again, more on how to use zigpy-cli to migrate by backing up to file and restoring to new adapter from file is discussed here -> #16478

PS: There is also an unofficial Zigpy-CLI add-on for Home Assistant that if plug in the adapter in same computer and stop apps accessing it.

https://community.home-assistant.io/t/new-add-on-for-running-zigpy-cli-commands-like-energy-scans/669727

https://github.com/tube0013/tubeszb_addons/tree/main/tzb-zigpy-cli-tools

https://github.com/tube0013/tubeszb_addons/blob/main/tzb-zigpy-cli-tools/DOCS.md

[dafunkydan]

@tommyjlong , Thanks again for your really detailled Documentation, highly appreciate your Time!
Following it, i was able to reproduce everything, and swap between both Sticks on the Livesystem! 👍

However, this is just Step One on my Mission:

My Goal is to have a Second spare Zigbee Dongle, ready to take over with a second spare RPi with HAOS, in Case of a Failure

Unfortunatly, it didn't work when using the Replacement-RPi.
After everything from your Write-Up was working, i did a Full HA-Backup of my Live-System, a full Restore on the Fallback-System ("Wipe and Restore"), and had to find out the Replacement-Stick doesn't work on that 😢

Z2Ms Log start with:

info  2024-01-18 15:31:15: Starting zigbee-herdsman (0.30.0)
info  2024-01-18 15:31:20: zigbee-herdsman started (resumed)
info  2024-01-18 15:31:20: Coordinator firmware version: '{"meta":{"maintrel":1,"majorrel":2,"minorrel":7,"product":1,"revision":20221226,"transportrev":2},"type":"zStack3x0"}'
info  2024-01-18 15:31:20: Currently 48 devices are joined:

Which looks alright, as the "Restoring" already took place on the Live System, which is the Source for the Backup used on the Fallback-System. However, no Routers are operatable: Data request failed with error: 'No network route'

This is the second Part i just don't get - it is a full restored Backup of the System where it was working prior to it.
Even funnier, after that failed, i went back to the Main System, which instantly worked with the Original Stick - but not longer with the Replacement Stick.

I really have no Idea what the... is going on and how i could track down the Problem. I have no Explanation how a fully restored Backup System with a working Replacement-Stick just fails. Any Ideas? @tommyjlong You don't have by accident a spare System as well, do you? 😝

[Hedda]

Did you write the same IEEE address to your new Zigbee Coordinator adapter as the IEEE address of your old adapter?

• https://www.zigbee2mqtt.io/guide/adapters/flashing/copy_ieeaddr.html
  • https://www.zigbee2mqtt.io/guide/adapters/flashing/copy_ieeaddr.html#zigstar-multi-tool
  • https://www.zigbee2mqtt.io/guide/adapters/flashing/copy_ieeaddr.html#cc2538-bsl
  • https://www.zigbee2mqtt.io/guide/adapters/flashing/copy_ieeaddr.html#flash-programmer-2

If you do not copy and write the same IEEE adders then you will have to re-pair all your devices:

https://www.zigbee2mqtt.io/guide/faq/#how-do-i-migrate-from-one-adapter-to-another

[tommyjlong]

Let me see if I have this correct....

1. the Replacement stick worked the first time in the Live/Main system. Then,
2. Backup on Live/Main and use the data to restore to RPi, moved Replacement stick to RPi where you get the router error, then
3. Activated Live/Main system with Original stick and things worked. Then
4. Moved Replacement stick back to the Live/Main system and you get route errors again.

Honestly, I don't know, but here is a long thread on the subject.
#12774

My best guess is that the other devices/routers in the network are seeing changes during all of this and the routes haven't had time to reorganized. I can especially see this if the RPi is in a different location. I've read where people suggest turning off the Coordinator for at least 20 minutes, and perhaps as long as 24 hours as a fix.

Another thing, is that an HA Backup/Restore of the Z2M AddOn is not the same as the Z2M backup/restore of the coordinator. From my testing, the HA Backup of the Z2M AddOn does not backup the database nor coordinator_backup.json file, so a HA restore of the AddOn will leave the RPi with empty Z2M data.

[dafunkydan]

Did you write the same IEEE address to your new Zigbee Coordinator adapter as the IEEE address of your old adapter?

Yes surely, see above

• the Replacement stick worked the first time in the Live/Main system. Then,

• Backup on Live/Main and use the data to restore to RPi, moved Replacement stick to RPi where you get the router error, then

• Activated Live/Main system with Original stick and things worked. Then

• Moved Replacement stick back to the Live/Main system and you get route errors again.

Yes, absolutely correct!

My best guess is that the other devices/routers in the network are seeing changes during all of this

While i can't be 100% sure, i rather doubt it, as it was working almost instantly in first place - and i did the same thing regulary for more than a year with a conbee2 stick on deconz, and never encountered such problems.

From my testing, the HA Backup of the Z2M AddOn does not backup the database nor coordinator_backup.json file

Well, on a sidenote, on the Restored RPi there are all Devices listed in Z2M, even those just added before making the backup on the Live system.
Anyway, the more i am testing, the more i suspect Z2M itself beeing the culprit. I think my next testings will be a combination of reflashing the replacement-Stick, and restoring the most recent Backups with TubesZB Zigpy-CLI Tools and/or zigpy-cli-tools.

I never thought this will become such a Rabbithole 😆
@tommyjlong and @Hedda as well, Thank you Guys so much for all your Input and Support, highly appreciate it!

[dafunkydan]

Now i tested many possible Combinations, and my Observings are:

Only the last restored* Stick can make a Connection.
I can easily swap the current active Stick between RPis without any significant waiting Time.
It doesn't matter on which System i restored the Stick, the other System will happily and (almost) instantly work with it.

Once the other Stick got restored*, the first one doesn't connect to the Network, on any of the RPis.

*restored = Flashed Firmware including setting the IEEE Adress. The latter might not be necessary.
Next Time Z2M starts with that flashed Stick, it gets restored.
And then this is the working Stick.

So - There is clearly something more, aside the IEEE Adress, and the (restored) Z2M Configuration.
The big Question is: WTH 😄
What prevents Z2M to work with a previously working Stick, the only difference beeing that the other Stick got restored on another System in the Meantime?

[dafunkydan]

@Koenkk I hope thats okay and you don't mind asking you directly if you have an Idea:
If i want to swap my Sticks (same Model, same IEEE Adress set) i need to reflash it first, in order to make Z2M restore the Settings at Startup, and operate my Network.
Is there a Way to force Z2M to restore at Startup, without reflashing the other Stick first?