[REQUEST] Zeroconf (mDNS) automatic network discovery compatibility with Home Assistant ZHA integration

[Hedda]

@thegroove Please consider adding automatic network discovery of this so can be discovered by Home Assistant’s ZHA integration:

https://community.home-assistant.io/t/zha-automatic-discovery-of-zigbee-coordinator-bridges-gateways-ethernet-wifi-network-devices-that-support-zeroconf-or-ssdp/293300

Support for Zeroconf network discovery of Tube's ESPHome based Zigbee Gateways was recently added to the ZHA integration:

home-assistant/core#48420

I believe that would need to explicitly add support for the mDNS record name that you choose to Home Assistant for discovery:

home-assistant/core@6ba0629

PS: I think that automatically discovering it makes this more user-friendly compared to tasmota-zbbridge on ITead Sonoff ZBBridge.

[thegroove]

I'll have to look into this, but at first glance, it seems logical to me to set up zeroconf for a generic EZSP serial gateway type, as in the end they're all the same from zha's perspective. This way, any ESPHome (or other, like Tasmota) device could advertise it, not just the Sonoff ZB bridge or Tube's gateways. There's already people using the stream.h version of the component with other random combinations of ESPHome and an efr32 radio, that work exactly the same but wouldn't advertise as a ZB bridge or tube gateway necessarily. In the end, all ZHA cares about is a serial endpoint that speaks EZSP, so that should be enough for a zeroconf implementation. I would also prefer the port to be a variable instead of fixed, if zeroconf allows it. No reason to dictate a specific port otherwise, as it would only complicate implementation on other setups.

At this point I don't know how ESPHome sets up mdns/zeroconf and how hard it is to tap into that from an external custom component, but I'll investigate.

[Hedda]

I'll have to look into this, but at first glance, it seems logical to me to set up zeroconf for a generic EZSP serial gateway type, as in the end they're all the same from zha's perspective.

Yes but as I understand it, what is different now with Tube's Zigbee Gateways that Home Assistant via the PR home-assistant/core#48420 now explicitly know that if discover a device via mDNS that specifically has the name "_tube_zb_gw._tcp.local." then it will pass that on to the ZHA integration which in turn will get the discovered type parameters, which include, port to use for the serial-to-IP gateway service, baud rate speed to use, and radio type, (noting that @tube0013 has both Texas Instruments Z-Stack ZNP based gateways as well as Silicon Labs EZSP based gateways so specific radio type in such cases must always be explicit).

home-assistant/core@6ba0629

I would also prefer the port to be a variable instead of fixed.

You can set any TCP port you like for the actual serial-to-IP gateway service, it is just the mDNS broadcast messages that are done on a fixed port for Zeroconf service discovery, that way all mDNS servers and clients always know which port to listen on:

https://en.wikipedia.org/wiki/Zero-configuration_networking#Service_discovery

https://en.wikipedia.org/wiki/Zero-configuration_networking#DNS-SD_with_multicast

"Each host listens on the mDNS port, 5353, transmitted to a well-known multicast address and resolves requests for the DNS record of its .local hostname (e.g. the A, AAAA, CNAME) to its IP address. When an mDNS client needs to resolve a local hostname to an IP address, it sends a DNS request for that name to the well-known multicast address; the computer with the corresponding A/AAAA record replies with its IP address. The mDNS multicast address is 224.0.0.251 for IPv4 and ff02::fb for IPv6 link-local addressing."

[Hedda]

At this point I don't know how ESPHome sets up mdns/zeroconf and how hard it is to tap into that from an external custom component, but I'll investigate.

I believe that Zeroconf/mDNS is enabled by default in ESPHome:

https://github.com/tube0013/tube_gateways/tree/main/esphome

So probably just need to make sure at least have "efr32" in the name for esphome on the ESPHome side for that? Example:

esphome:
  name: esphome_zbbridge_ezsp_efr32

[tube0013]

It's based off the name of the device

[thegroove]

Is that a zeroconf requirement or is that just the way it's currently done? Name of the device is an iffy thing to use as a matching criterium, as we're talking services here, not necessarily devices. But I need to dive into the workings of zeroconf before I can say anything sensible about it.

[tube0013]

It is the way it's currently done. The ZHA discovery piggy backs on the esphome mDNS. As I'm flashing the modules I am selling they are named accordingly, and thus are explicitly supported by discovery.

[thegroove]

I see there's MDNS.addService() and MDNS.addServiceTxt() methods that are available in global scope, will test if I can safely call those from the component.

[Hedda]

It's based off the name of the device

@tube0013 Maybe bikeshedding but could I suggest that HA is changed so as not explicitly use "efr32" to find the radio type?

That is, I think that it would be much more logical to explicitly use "ezsp" and "znp" respectivly for Silabs and TI radio types:

Radio Type	Zigbee Radio Hardware
ezsp	Silicon Labs EmberZNet protocol (e.g., Elelabs, HUSBZB-1, Telegesis)
deconz	dresden elektronik deCONZ protocol (e.g., ConBee I/II, RaspBee I/II)
znp	Texas Instruments new (active): Z-Stack 3.x.x ZNP protocol (e.g., CC26x2, CC13x2)
zigate	ZiGate Serial protocol (e.g., ZiGate USB-TTL, PiZiGate, ZiGate WiFi)
xbee	Digi XBee ZB Coordinator Firmware protocol (e.g., Digi XBee Series 2, 2C, 3)

Regardless I think that Home Assistant should explicitly use "ezsp" and "znp", etc. as part of its ZHA discovery of radio type.

@thegroove To be backwards and forward compatible with that and the method used today you could choose to use a name like "esphome_zbbridge_ezsp_efr32" as that then obviously include both "ezsp" and "efr32" so it covers both those explicit names.

esphome:
  name: esphome_zbbridge_ezsp_efr32

Tube's Gateways would if so have to change naming to add "ezsp" in his name be future compatible with that explicit discovery:

https://github.com/tube0013/tube_gateways/blob/3535cd979ee058888cc7551321613f38038f1f89/esphome/tube_zb_gw_efr32_Series2.yaml

https://github.com/tube0013/tube_gateways/blob/3535cd979ee058888cc7551321613f38038f1f89/esphome/tube_zb_gw_efr32_Pro.yaml

Example change from "tube_zb_gw_efr32" to "tube_zb_gw_ezsp_efr32" to get *ezsp in the name for "ezsp" as radio type:

esphome:
  name: tube_zb_gw_efr32
  platform: ESP32
  board: esp-wrover-kit

https://github.com/tube0013/tube_gateways/blob/3535cd979ee058888cc7551321613f38038f1f89/esphome/tube_zb_gw_cc2652p2.yaml

Example change from "tube_zb_gw_cc2652p2" to "tube_zb_gw_znp_cc2652p2" to get *znp in the name for "znp" as radio type:

esphome:
  name: tube_zb_gw_znp_cc2652p2
  platform: ESP32
  board: esp-wrover-kit

If so then tube also need to update

https://github.com/tube0013/tube_gateways/blob/aa70e86ff73285d832e35745442de1d288433e3c/README.md

https://github.com/tube0013/tube_gateways/blob/3535cd979ee058888cc7551321613f38038f1f89/esphome/README.md

[Hedda]

Is that a zeroconf requirement or is that just the way it's currently done? Name of the device is an iffy thing to use as a matching criterium, as we're talking services here, not necessarily devices. But I need to dive into the workings of zeroconf before I can say anything sensible about it.

As tube noted; using the device name for explicit discovery was is initial method used, but "TXT record" via DNS-SD could be used?

Zeroconf DNS-SD (DNS Service Discovery) should be able to pass additional attributes such as config parameters via "TXT records".

https://en.wikipedia.org/wiki/Zero-configuration_networking#DNS-based_service_discovery

"Each service instance is described using a DNS SRV and DNS TXT record. A client discovers the list of available instances for a given service type by querying the DNS PTR record of that service type's name; the server returns zero or more names of the form ., each corresponding to a SRV/TXT record pair. The SRV record resolves to the domain name providing the instance, while the TXT can contain service-specific configuration parameters."

https://en.wikipedia.org/wiki/Zero-configuration_networking#DNS-SD_with_multicast

"DNS Service Discovery (DNS-SD) requests can also be sent using mDNS to yield zero-configuration DNS-SD. This uses DNS PTR, SRV, TXT records to advertise instances of service types, domain names for those instances, and optional configuration parameters for connecting to those instances. But SRV records can now resolve to .local domain names, which mDNS can resolve to local IP addresses."

https://en.wikipedia.org/wiki/TXT_record

"TXT record (short for text record) is a type of resource record in the Domain name system (DNS) used to provide the ability to associate arbitrary text with a host or other name, such as human readable information about a server, network, data center, or other accounting information. It is also often used in a more structured fashion to record small amounts of machine-readable data into the DNS."

[Hedda]

DNS-SD (Rendezvous) TXT record format -> http://www.zeroconf.org/Rendezvous/txtrecords.html

Example TXT record containing three name/value pairs

---

| 0x0A | name=value | 0x08 | paper=A4 | 0x12 | Rendezvous Is Cool |

[Hedda]

If possible what information should Zeroconf DNS-SD pass along to the ZHA integration via TXT record?

• radio_type=ezsp
• baud_rate=value
• data_flow_control=software
• tcp_port_serial_gateway=value

[thegroove]

= enp7s0 IPv4 sonoff_zb_bridge                              _ezsp._tcp           local
hostname = [sonoff_zb_bridge.local]
                                                                                                        address = [172.16.0.249]
                                                                                                                     port = [6638]
                                                                                                                                  - txt = []

I've managed to get esphome to advertise through mdns. Service name here is 'ezsp'. The txt=[] field is where the extra value pairs go. The port is part of the service definition, no need to add that through txt. I'm playing with adding the extra information through the txt field, e.g. "baud_rate=115200" I can already add (though I'm struggling a bit to get the actual baud rate from esphome without duplication, but I'll figure that out).
I'm in doubt about the flow control parameter. I can easily add it as a static field, but esphome has no concept of (hardware) flow control and it's not set anywhere in the serial bridging code. Does hw flow control even apply over tcp / would zigpy - zha support it?
No flow control vs software flow control is dependent on the zb firmware and the client (i.e. zha), the serial layer is unaware of it, so it would annoy me to have to add it to the serial server. I'd probably split out the mdns part in my component if it were a requirement for proper integration with zha, but if we definitely need that information, it's not a problem -- just want to confirm before I do the extra work.

[thegroove]

Been thinking about this some more and realized I've also contaminated the generic serial server code with "ezsp" already, so I'm going to separate the mdns service announcement anyhow to get rid of that. Still would be good to settle on what to do with the flow control attribute though, I'm always in favor of keeping things as simple as possible, so if we don't necessarily need it, my preference is to simply ignore it and let HA/ZHA figure it out.

[Hedda]

No flow control vs software flow control is dependent on the zb firmware and the client (i.e. zha), the serial layer is unaware of it, so it would annoy me to have to add it to the serial server.

While the serial server does not need that, ZHA does need to know flow control method used for it to properly communicate with the zigbee firmware. That is, if the zigbee firmware has software flow control set then ZHA config flow needs to know that.

Still would be good to settle on what to do with the flow control attribute though, I'm always in favor of keeping things as simple as possible, so if we don't necessarily need it, my preference is to simply ignore it and let HA/ZHA figure it out.

ZHA and the radio libraries for zigpy does not have a probe that will negotiate to figure it out any of those settings for you.

• radio_type=ezsp
• baud_rate_speed=115200
• data_flow_control=software
• tcp_port_serial_gateway=8888

[Hedda]

FYI, Home Assistant's zeroconf maintainer/dev @bdraco referred to this example for using TXT record to pass values to integrations:

https://github.com/home-assistant/core/blob/d4ed65e0f53fcab991b13061b1101470f24287a6/homeassistant/components/homekit_controller/config_flow.py#L199

[Hedda]

By the way, alos see this discussion regarding possible flow control parameters in ESPHome-> esphome/feature-requests#1126

[Hedda]

FYI, Tube's Zigbee Gateways are now (only) device listed in ZHA docs as discoverable via Zeroconf, see:

https://www.home-assistant.io/integrations/zha/#discovery-via-usb-or-zeroconf

More discussion here:

https://community.home-assistant.io/t/zha-automatic-discovery-of-zigbee-coordinator-bridges-gateways-ethernet-wifi-network-devices-that-support-zeroconf-or-ssdp/293300/

[Hedda]

FYI, Zero-configuration networking (zeroconf) looks to have been changed and improved in the Home Assistant 2022.02 release:

home-assistant/core#62133

https://www.home-assistant.io/integrations/zeroconf/

This change is only a concern for custom integration developers.

Currently zeroconf matching only allows matching the macaddress, model, and manufacturer properties along with the name from the ZeroconfServiceInfo.

Since properties are arbitrarily defined by the zeroconf service, the list of named properties has grown over time.

Matching now allows for any arbitrarily defined property. All property matches must be lowercase, wildcards are supported

The top level keys model, manufacturer, and macaddress are now deprecated from components manifest.json file and should be moved into a properties dict.

For example:

-    {"type":"_airplay._tcp.local.","model":"appletv*"}
+    {"type":"_airplay._tcp.local.","properties":{"model":"appletv*"}}

[Hedda]

By the way, Home Assistant founders/leads now explicitly said this year they will prioritize focus on features like these types:

https://www.home-assistant.io/blog/2022/01/19/streamlining-experiences/

Check out this video at around 11 min 55 sec in where talk about that focus point about making it easier to get started, etc.

https://www.youtube.com/watch?v=t_2D_KoFIfU&t=710s

Hopefully, it should be a greater chance of also getting more help from other Home Assistant developers with this if ask them.

[Hedda]

FYI, PR for ZiGate's Zeroconf has been merged in Home Assistant core making it an example for adding additional more readable:

home-assistant/core#68577

https://github.com/home-assistant/core/blob/dev/homeassistant/components/zha/manifest.json#L85-L88

https://github.com/home-assistant/core/blob/dev/homeassistant/components/zha/config_flow.py#L198-L201

https://github.com/home-assistant/core/blob/dev/tests/components/zha/test_config_flow.py#L80-L107

https://github.com/home-assistant/core/blob/dev/homeassistant/generated/zeroconf.py#L384-L389