Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ZHA becomes unavailable. Fails to trigger lights or other zigbee devices #99305

Closed
mathiasolofsson87 opened this issue Aug 29, 2023 · 117 comments
Closed
Assignees

Comments

@mathiasolofsson87
Copy link

The problem

Randomly ZHA stops being able to send requests to zigbee devices and reports back EmberStatus.delivery FAILED on for example light.turn on or off service. This can start happening after a few minutes of a restart, or after a week of running smoothly. I was finally able to find info in the log when this starts happening. I believe this started happening around 2023.7.x.

Only Zigbee lights or devices are affected, Wifi lights or other wifi devices work fine.

What version of Home Assistant Core has the issue?

2023.8.4

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

ZHA

Link to integration documentation on our website

https://www.home-assistant.io/integrations/zha/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

2023-08-29 22:40:21.491 ERROR (MainThread) [homeassistant.core] Error executing service: <ServiceCall light.turn_on (c:01H91GD6XT571DEY5HRT5WJC8P): entity_id=['light.tv_banken_play_1_huelight_2'], params=transition=30.0, brightness=91, color_temp=454, color_temp_kelvin=2202>
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 64, in wrapper
    return await RETRYABLE_REQUEST_DECORATOR(func)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/util.py", line 132, in retry
    return await func()
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/zcl/__init__.py", line 375, in request
    return await self._endpoint.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/endpoint.py", line 253, in request
    return await self.device.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/device.py", line 293, in request
    await self._application.request(
  File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 824, in request
    await self.send_packet(
  File "/usr/local/lib/python3.11/site-packages/bellows/zigbee/application.py", line 831, in send_packet
    raise zigpy.exceptions.DeliveryError(
zigpy.exceptions.DeliveryError: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/core.py", line 1990, in _run_service_call_catch_exceptions
    await coro_or_task
  File "/usr/src/homeassistant/homeassistant/core.py", line 2011, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 235, in handle_service
    return await service.entity_service_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 870, in entity_service_call
    response_data = await _handle_entity_call(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 942, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 580, in async_handle_light_on_service
    await light.async_turn_on(**filter_turn_on_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/zha/light.py", line 336, in async_turn_on
    result = await self._level_cluster_handler.move_to_level_with_on_off(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 75, in wrapper
    raise HomeAssistantError(message) from exc
homeassistant.exceptions.HomeAssistantError: Failed to send request: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>
2023-08-29 22:40:21.502 ERROR (MainThread) [homeassistant.core] Error executing service: <ServiceCall light.turn_on (c:01H91GD6XT5A12931RCK0S6AJ1): entity_id=['light.tv_banken_play_2_huelight'], params=transition=30.0, brightness=91, color_temp=454, color_temp_kelvin=2202>
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 64, in wrapper
    return await RETRYABLE_REQUEST_DECORATOR(func)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/util.py", line 132, in retry
    return await func()
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/zcl/__init__.py", line 375, in request
    return await self._endpoint.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/endpoint.py", line 253, in request
    return await self.device.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/device.py", line 293, in request
    await self._application.request(
  File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 824, in request
    await self.send_packet(
  File "/usr/local/lib/python3.11/site-packages/bellows/zigbee/application.py", line 831, in send_packet
    raise zigpy.exceptions.DeliveryError(
zigpy.exceptions.DeliveryError: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/core.py", line 1990, in _run_service_call_catch_exceptions
    await coro_or_task
  File "/usr/src/homeassistant/homeassistant/core.py", line 2011, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 235, in handle_service
    return await service.entity_service_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 870, in entity_service_call
    response_data = await _handle_entity_call(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 942, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 580, in async_handle_light_on_service
    await light.async_turn_on(**filter_turn_on_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/zha/light.py", line 336, in async_turn_on
    result = await self._level_cluster_handler.move_to_level_with_on_off(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 75, in wrapper
    raise HomeAssistantError(message) from exc
homeassistant.exceptions.HomeAssistantError: Failed to send request: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>
2023-08-29 22:40:21.513 ERROR (MainThread) [homeassistant.core] Error executing service: <ServiceCall light.turn_on (c:01H91GD6XTRE80T0Y8M28NESHG): entity_id=['light.tv_banken_light'], params=transition=30.0, brightness=91, color_temp=454, color_temp_kelvin=2202>
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 64, in wrapper
    return await RETRYABLE_REQUEST_DECORATOR(func)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/util.py", line 132, in retry
    return await func()
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/zcl/__init__.py", line 375, in request
    return await self._endpoint.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/endpoint.py", line 253, in request
    return await self.device.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/device.py", line 293, in request
    await self._application.request(
  File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 824, in request
    await self.send_packet(
  File "/usr/local/lib/python3.11/site-packages/bellows/zigbee/application.py", line 831, in send_packet
    raise zigpy.exceptions.DeliveryError(
zigpy.exceptions.DeliveryError: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/core.py", line 1990, in _run_service_call_catch_exceptions
    await coro_or_task
  File "/usr/src/homeassistant/homeassistant/core.py", line 2011, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 235, in handle_service
    return await service.entity_service_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 870, in entity_service_call
    response_data = await _handle_entity_call(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 942, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 580, in async_handle_light_on_service
    await light.async_turn_on(**filter_turn_on_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/zha/light.py", line 336, in async_turn_on
    result = await self._level_cluster_handler.move_to_level_with_on_off(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 75, in wrapper
    raise HomeAssistantError(message) from exc
homeassistant.exceptions.HomeAssistantError: Failed to send request: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>
2023-08-29 22:40:21.522 ERROR (MainThread) [homeassistant.core] Error executing service: <ServiceCall light.turn_on (c:01H91GD6XT7DJ1Z6N9VVQH2DCS): entity_id=['light.fonster_3_light'], params=transition=30.0, brightness=91, color_temp=454, color_temp_kelvin=2202>
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 64, in wrapper
    return await RETRYABLE_REQUEST_DECORATOR(func)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/util.py", line 132, in retry
    return await func()
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/zcl/__init__.py", line 375, in request
    return await self._endpoint.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/endpoint.py", line 253, in request
    return await self.device.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/device.py", line 293, in request
    await self._application.request(
  File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 824, in request
    await self.send_packet(
  File "/usr/local/lib/python3.11/site-packages/bellows/zigbee/application.py", line 831, in send_packet
    raise zigpy.exceptions.DeliveryError(
zigpy.exceptions.DeliveryError: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/core.py", line 1990, in _run_service_call_catch_exceptions
    await coro_or_task
  File "/usr/src/homeassistant/homeassistant/core.py", line 2011, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 235, in handle_service
    return await service.entity_service_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 870, in entity_service_call
    response_data = await _handle_entity_call(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 942, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 580, in async_handle_light_on_service
    await light.async_turn_on(**filter_turn_on_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/zha/light.py", line 336, in async_turn_on
    result = await self._level_cluster_handler.move_to_level_with_on_off(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 75, in wrapper
    raise HomeAssistantError(message) from exc
homeassistant.exceptions.HomeAssistantError: Failed to send request: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>
2023-08-29 22:40:21.533 ERROR (MainThread) [homeassistant.core] Error executing service: <ServiceCall light.turn_on (c:01H91GD6XVF4CR39YGJZ9B3M3C): entity_id=['light.fonster_2_light_2'], params=transition=30.0, brightness=91, color_temp=454, color_temp_kelvin=2202>
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 64, in wrapper
    return await RETRYABLE_REQUEST_DECORATOR(func)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/util.py", line 132, in retry
    return await func()
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/zcl/__init__.py", line 375, in request
    return await self._endpoint.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/endpoint.py", line 253, in request
    return await self.device.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/device.py", line 293, in request
    await self._application.request(
  File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 824, in request
    await self.send_packet(
  File "/usr/local/lib/python3.11/site-packages/bellows/zigbee/application.py", line 831, in send_packet
    raise zigpy.exceptions.DeliveryError(
zigpy.exceptions.DeliveryError: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/core.py", line 1990, in _run_service_call_catch_exceptions
    await coro_or_task
  File "/usr/src/homeassistant/homeassistant/core.py", line 2011, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 235, in handle_service
    return await service.entity_service_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 870, in entity_service_call
    response_data = await _handle_entity_call(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 942, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 580, in async_handle_light_on_service
    await light.async_turn_on(**filter_turn_on_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/zha/light.py", line 336, in async_turn_on
    result = await self._level_cluster_handler.move_to_level_with_on_off(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 75, in wrapper
    raise HomeAssistantError(message) from exc
homeassistant.exceptions.HomeAssistantError: Failed to send request: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>
2023-08-29 22:40:21.544 ERROR (MainThread) [homeassistant.core] Error executing service: <ServiceCall light.turn_on (c:01H91GD6XV18HC4PNRW52VNN0M): entity_id=['light.fonster_1_light'], params=transition=30.0, brightness=91, color_temp=454, color_temp_kelvin=2202>
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 64, in wrapper
    return await RETRYABLE_REQUEST_DECORATOR(func)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/util.py", line 132, in retry
    return await func()
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/zcl/__init__.py", line 375, in request
    return await self._endpoint.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/endpoint.py", line 253, in request
    return await self.device.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/device.py", line 293, in request
    await self._application.request(
  File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 824, in request
    await self.send_packet(
  File "/usr/local/lib/python3.11/site-packages/bellows/zigbee/application.py", line 831, in send_packet
    raise zigpy.exceptions.DeliveryError(
zigpy.exceptions.DeliveryError: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/core.py", line 1990, in _run_service_call_catch_exceptions
    await coro_or_task
  File "/usr/src/homeassistant/homeassistant/core.py", line 2011, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 235, in handle_service
    return await service.entity_service_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 870, in entity_service_call
    response_data = await _handle_entity_call(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 942, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 580, in async_handle_light_on_service
    await light.async_turn_on(**filter_turn_on_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/zha/light.py", line 336, in async_turn_on
    result = await self._level_cluster_handler.move_to_level_with_on_off(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 75, in wrapper
    raise HomeAssistantError(message) from exc
homeassistant.exceptions.HomeAssistantError: Failed to send request: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>
2023-08-29 22:40:21.553 ERROR (MainThread) [homeassistant.core] Error executing service: <ServiceCall light.turn_on (c:01H91GD6XV2RBGKC9RVJDTH6TE): entity_id=['light.krakan_light_2'], params=transition=30.0, brightness=91, color_temp=454, color_temp_kelvin=2202>
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 64, in wrapper
    return await RETRYABLE_REQUEST_DECORATOR(func)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/util.py", line 132, in retry
    return await func()
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/zcl/__init__.py", line 375, in request
    return await self._endpoint.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/endpoint.py", line 253, in request
    return await self.device.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/device.py", line 293, in request
    await self._application.request(
  File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 824, in request
    await self.send_packet(
  File "/usr/local/lib/python3.11/site-packages/bellows/zigbee/application.py", line 831, in send_packet
    raise zigpy.exceptions.DeliveryError(
zigpy.exceptions.DeliveryError: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/core.py", line 1990, in _run_service_call_catch_exceptions
    await coro_or_task
  File "/usr/src/homeassistant/homeassistant/core.py", line 2011, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 235, in handle_service
    return await service.entity_service_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 870, in entity_service_call
    response_data = await _handle_entity_call(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 942, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 580, in async_handle_light_on_service
    await light.async_turn_on(**filter_turn_on_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/zha/light.py", line 336, in async_turn_on
    result = await self._level_cluster_handler.move_to_level_with_on_off(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 75, in wrapper
    raise HomeAssistantError(message) from exc
homeassistant.exceptions.HomeAssistantError: Failed to send request: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>
2023-08-29 22:40:21.563 ERROR (MainThread) [homeassistant.core] Error executing service: <ServiceCall light.turn_on (c:01H91GD6XVK1SHG7C3JP0K0YFG): entity_id=['light.lekrummet_fonster_2_light'], params=transition=30.0, brightness=91, color_temp=454, color_temp_kelvin=2202>
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 64, in wrapper
    return await RETRYABLE_REQUEST_DECORATOR(func)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/util.py", line 132, in retry
    return await func()
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/zcl/__init__.py", line 375, in request
    return await self._endpoint.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/endpoint.py", line 253, in request
    return await self.device.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/device.py", line 293, in request
    await self._application.request(
  File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 824, in request
    await self.send_packet(
  File "/usr/local/lib/python3.11/site-packages/bellows/zigbee/application.py", line 831, in send_packet
    raise zigpy.exceptions.DeliveryError(
zigpy.exceptions.DeliveryError: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/core.py", line 1990, in _run_service_call_catch_exceptions
    await coro_or_task
  File "/usr/src/homeassistant/homeassistant/core.py", line 2011, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 235, in handle_service
    return await service.entity_service_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 870, in entity_service_call
    response_data = await _handle_entity_call(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 942, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 580, in async_handle_light_on_service
    await light.async_turn_on(**filter_turn_on_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/zha/light.py", line 336, in async_turn_on
    result = await self._level_cluster_handler.move_to_level_with_on_off(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 75, in wrapper
    raise HomeAssistantError(message) from exc
homeassistant.exceptions.HomeAssistantError: Failed to send request: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>
2023-08-29 22:40:21.573 ERROR (MainThread) [homeassistant.core] Error executing service: <ServiceCall light.turn_on (c:01H91GD6XVDTVDC6G34GN7T1VE): entity_id=['light.koket_fonster_light'], params=transition=30.0, brightness=91, color_temp=454, color_temp_kelvin=2202>
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 64, in wrapper
    return await RETRYABLE_REQUEST_DECORATOR(func)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/util.py", line 132, in retry
    return await func()
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/zcl/__init__.py", line 375, in request
    return await self._endpoint.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/endpoint.py", line 253, in request
    return await self.device.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/device.py", line 293, in request
    await self._application.request(
  File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 824, in request
    await self.send_packet(
  File "/usr/local/lib/python3.11/site-packages/bellows/zigbee/application.py", line 831, in send_packet
    raise zigpy.exceptions.DeliveryError(
zigpy.exceptions.DeliveryError: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/core.py", line 1990, in _run_service_call_catch_exceptions
    await coro_or_task
  File "/usr/src/homeassistant/homeassistant/core.py", line 2011, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 235, in handle_service
    return await service.entity_service_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 870, in entity_service_call
    response_data = await _handle_entity_call(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 942, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 580, in async_handle_light_on_service
    await light.async_turn_on(**filter_turn_on_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/zha/light.py", line 336, in async_turn_on
    result = await self._level_cluster_handler.move_to_level_with_on_off(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 75, in wrapper
    raise HomeAssistantError(message) from exc
homeassistant.exceptions.HomeAssistantError: Failed to send request: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>
2023-08-29 22:40:21.582 ERROR (MainThread) [homeassistant.core] Error executing service: <ServiceCall light.turn_on (c:01H91GD6XVK7030HR6Y6GXFNV6): entity_id=['light.hallen_vagg_ner_light_2'], params=transition=30.0, brightness=91, color_temp=454, color_temp_kelvin=2202>
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 64, in wrapper
    return await RETRYABLE_REQUEST_DECORATOR(func)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/util.py", line 132, in retry
    return await func()
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/zcl/__init__.py", line 375, in request
    return await self._endpoint.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/endpoint.py", line 253, in request
    return await self.device.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/device.py", line 293, in request
    await self._application.request(
  File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 824, in request
    await self.send_packet(
  File "/usr/local/lib/python3.11/site-packages/bellows/zigbee/application.py", line 831, in send_packet
    raise zigpy.exceptions.DeliveryError(
zigpy.exceptions.DeliveryError: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/core.py", line 1990, in _run_service_call_catch_exceptions
    await coro_or_task
  File "/usr/src/homeassistant/homeassistant/core.py", line 2011, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 235, in handle_service
    return await service.entity_service_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 870, in entity_service_call
    response_data = await _handle_entity_call(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 942, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 580, in async_handle_light_on_service
    await light.async_turn_on(**filter_turn_on_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/zha/light.py", line 336, in async_turn_on
    result = await self._level_cluster_handler.move_to_level_with_on_off(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 75, in wrapper
    raise HomeAssistantError(message) from exc
homeassistant.exceptions.HomeAssistantError: Failed to send request: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>
2023-08-29 22:40:21.592 ERROR (MainThread) [homeassistant.core] Error executing service: <ServiceCall light.turn_on (c:01H91GD6XV4M6QNPJPD85HARKP): entity_id=['light.hallen_vagg_upp_light'], params=transition=30.0, brightness=91, color_temp=454, color_temp_kelvin=2202>
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 64, in wrapper
    return await RETRYABLE_REQUEST_DECORATOR(func)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/util.py", line 132, in retry
    return await func()
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/zcl/__init__.py", line 375, in request
    return await self._endpoint.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/endpoint.py", line 253, in request
    return await self.device.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/device.py", line 293, in request
    await self._application.request(
  File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 824, in request
    await self.send_packet(
  File "/usr/local/lib/python3.11/site-packages/bellows/zigbee/application.py", line 831, in send_packet
    raise zigpy.exceptions.DeliveryError(
zigpy.exceptions.DeliveryError: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/core.py", line 1990, in _run_service_call_catch_exceptions
    await coro_or_task
  File "/usr/src/homeassistant/homeassistant/core.py", line 2011, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 235, in handle_service
    return await service.entity_service_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 870, in entity_service_call
    response_data = await _handle_entity_call(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 942, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 580, in async_handle_light_on_service
    await light.async_turn_on(**filter_turn_on_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/zha/light.py", line 336, in async_turn_on
    result = await self._level_cluster_handler.move_to_level_with_on_off(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 75, in wrapper
    raise HomeAssistantError(message) from exc
homeassistant.exceptions.HomeAssistantError: Failed to send request: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>
2023-08-29 22:40:21.602 ERROR (MainThread) [homeassistant.core] Error executing service: <ServiceCall light.turn_on (c:01H91GD6XVJQHEYXAJNQTA3KDS): entity_id=['light.trappen_fonster_light'], params=transition=30.0, brightness=91, color_temp=454, color_temp_kelvin=2202>
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 64, in wrapper
    return await RETRYABLE_REQUEST_DECORATOR(func)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/util.py", line 132, in retry
    return await func()
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/zcl/__init__.py", line 375, in request
    return await self._endpoint.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/endpoint.py", line 253, in request
    return await self.device.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/device.py", line 293, in request
    await self._application.request(
  File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 824, in request
    await self.send_packet(
  File "/usr/local/lib/python3.11/site-packages/bellows/zigbee/application.py", line 831, in send_packet
    raise zigpy.exceptions.DeliveryError(
zigpy.exceptions.DeliveryError: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/core.py", line 1990, in _run_service_call_catch_exceptions
    await coro_or_task
  File "/usr/src/homeassistant/homeassistant/core.py", line 2011, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 235, in handle_service
    return await service.entity_service_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 870, in entity_service_call
    response_data = await _handle_entity_call(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 942, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 580, in async_handle_light_on_service
    await light.async_turn_on(**filter_turn_on_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/zha/light.py", line 336, in async_turn_on
    result = await self._level_cluster_handler.move_to_level_with_on_off(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 75, in wrapper
    raise HomeAssistantError(message) from exc
homeassistant.exceptions.HomeAssistantError: Failed to send request: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>
2023-08-29 22:40:21.612 ERROR (MainThread) [homeassistant.core] Error executing service: <ServiceCall light.turn_on (c:01H91GD6XWC6FA4FY4GFKF5E3G): entity_id=['light.ovanvaningen_fonster_light'], params=transition=30.0, brightness=91>
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 64, in wrapper
    return await RETRYABLE_REQUEST_DECORATOR(func)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/util.py", line 132, in retry
    return await func()
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/zcl/__init__.py", line 375, in request
    return await self._endpoint.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/endpoint.py", line 253, in request
    return await self.device.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/device.py", line 293, in request
    await self._application.request(
  File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 824, in request
    await self.send_packet(
  File "/usr/local/lib/python3.11/site-packages/bellows/zigbee/application.py", line 831, in send_packet
    raise zigpy.exceptions.DeliveryError(
zigpy.exceptions.DeliveryError: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/core.py", line 1990, in _run_service_call_catch_exceptions
    await coro_or_task
  File "/usr/src/homeassistant/homeassistant/core.py", line 2011, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 235, in handle_service
    return await service.entity_service_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 870, in entity_service_call
    response_data = await _handle_entity_call(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 942, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 580, in async_handle_light_on_service
    await light.async_turn_on(**filter_turn_on_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/zha/light.py", line 336, in async_turn_on
    result = await self._level_cluster_handler.move_to_level_with_on_off(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 75, in wrapper
    raise HomeAssistantError(message) from exc
homeassistant.exceptions.HomeAssistantError: Failed to send request: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>

Additional information

No response

@home-assistant
Copy link

Hey there @dmulcahey, @Adminiuga, @puddly, mind taking a look at this issue as it has been labeled with an integration (zha) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of zha can trigger bot actions by commenting:

  • @home-assistant close Closes the issue.
  • @home-assistant rename Awesome new title Renames the issue.
  • @home-assistant reopen Reopen the issue.
  • @home-assistant unassign zha Removes the current integration label and assignees on the issue, add the integration domain after the command.

(message by CodeOwnersMention)


zha documentation
zha source
(message by IssueLinks)

@puddly
Copy link
Contributor

puddly commented Aug 29, 2023

Please include the diagnostics JSON for the ZHA integration.

A delivery failure after three retries is just that: a delivery failure. It's not something that can really be fixed within ZHA because individual device connectivity is entirely controlled by your mesh and outside of ZHA's control.

@mathiasolofsson87
Copy link
Author

Hi ok, it's just weird that after an update i've begun experiencing very frequent delivery failures where before the zigbee mesh was solid and stable. Attaching Diagnostics JSON here.

config_entry-zha-09a1032b0f8561eb3275b8b85206994d.json.txt

@GuyLewin
Copy link

Hi ok, it's just weird that after an update i've begun experiencing very frequent delivery failures where before the zigbee mesh was solid and stable. Attaching Diagnostics JSON here.

config_entry-zha-09a1032b0f8561eb3275b8b85206994d.json.txt

Same here. ZHA was working smoothly until this most recent version. Now I'm having issues losing control over lights on a daily basis.

@somerandomuser1
Copy link

same here. for me its a door sensor and temp sensor. they are solid and stable for like 2-3 days, then they go unavailable. Sometimes I get lucky and can Re-Pair them and they will work again for a day or 2, sometimes they wont even Re-Pair and I have to delete the entire network, readd everything from scratch.

@mathiasolofsson87
Copy link
Author

I usually dont have to do anything, eventually stability comes back and lights work again. or I just reload the ZHA integration and everything works again for a random amount of time

@puddly
Copy link
Contributor

puddly commented Aug 30, 2023

@GuyLewin @somerandomuser1 can you edit your comments to include the same diagnostic information for the ZHA integration (not individual devices, the integration itself!)?

@SNIPER722
Copy link

SNIPER722 commented Aug 30, 2023

same issue with home assistant yellow. multiple lumi device become unavailable
aqaura AVOC
zha-22cdc6274f48e6964f5877d0eddd7635-LUMI lumi.airmonitor.acn01-9db7ca02677341695a67ad2b2983afa6.json.txt
aqaura motion
zha-22cdc6274f48e6964f5877d0eddd7635-LUMI lumi.motion.ac02-adc5022c733fa0c0594ebdb8dea34467.json.txt
aqaura contact
zha-22cdc6274f48e6964f5877d0eddd7635-LUMI lumi.magnet.acn001-697076c6eee5982555d29deb76c9cf43.json.txt

attempt to reconfigure:
image

with error log:
`The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/zha/core/device.py", line 574, in async_configure
await self.identify_ch.trigger_effect(
File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/init.py", line 75, in wrapper
raise HomeAssistantError(message) from exc
homeassistant.exceptions.HomeAssistantError: Failed to send request: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>
2023-08-30 15:18:06.424 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/init.py", line 64, in wrapper
return await RETRYABLE_REQUEST_DECORATOR(func)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/zigpy/util.py", line 132, in retry
return await func()
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/zigpy/zcl/init.py", line 375, in request
return await self._endpoint.request(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/zigpy/endpoint.py", line 253, in request
return await self.device.request(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/zigpy/device.py", line 293, in request
await self._application.request(
File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 824, in request
await self.send_packet(
File "/usr/local/lib/python3.11/site-packages/bellows/zigbee/application.py", line 831, in send_packet
raise zigpy.exceptions.DeliveryError(
zigpy.exceptions.DeliveryError: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/zha/core/device.py", line 574, in async_configure
await self.identify_ch.trigger_effect(
File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/init.py", line 75, in wrapper
raise HomeAssistantError(message) from exc
homeassistant.exceptions.HomeAssistantError: Failed to send request: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>`

@Adminiuga
Copy link
Contributor

same issue with home assistant yellow. multiple lumi device become unavailable
aqaura AVOC
zha-22cdc6274f48e6964f5877d0eddd7635-LUMI lumi.airmonitor.acn01-9db7ca02677341695a67ad2b2983afa6.json.txt
aqaura motion
zha-22cdc6274f48e6964f5877d0eddd7635-LUMI lumi.motion.ac02-adc5022c733fa0c0594ebdb8dea34467.json.txt
aqaura contact
zha-22cdc6274f48e6964f5877d0eddd7635-LUMI lumi.magnet.acn001-697076c6eee5982555d29deb76c9cf43.json.txt

attempt to reconfigure:
image

with error log:
`The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/zha/core/device.py", line 574, in async_configure
await self.identify_ch.trigger_effect(
File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/init.py", line 75, in wrapper
raise HomeAssistantError(message) from exc
homeassistant.exceptions.HomeAssistantError: Failed to send request: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>
2023-08-30 15:18:06.424 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/init.py", line 64, in wrapper
return await RETRYABLE_REQUEST_DECORATOR(func)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/zigpy/util.py", line 132, in retry
return await func()
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/zigpy/zcl/init.py", line 375, in request
return await self._endpoint.request(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/zigpy/endpoint.py", line 253, in request
return await self.device.request(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/zigpy/device.py", line 293, in request
await self._application.request(
File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 824, in request
await self.send_packet(
File "/usr/local/lib/python3.11/site-packages/bellows/zigbee/application.py", line 831, in send_packet
raise zigpy.exceptions.DeliveryError(
zigpy.exceptions.DeliveryError: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/zha/core/device.py", line 574, in async_configure
await self.identify_ch.trigger_effect(
File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/init.py", line 75, in wrapper
raise HomeAssistantError(message) from exc
homeassistant.exceptions.HomeAssistantError: Failed to send request: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>`

Aqara/Lumi devices require a "compatible" router device as a parent, otherwise those are dropped from the network and do not reconnect. There was a list of compatible aqara routers posted on forums.

@SNIPER722
Copy link

SNIPER722 commented Aug 30, 2023

Aqara/Lumi devices require a "compatible" router device as a parent, otherwise those are dropped from the network and do not reconnect. There was a list of compatible aqara routers posted on forums.

that is something I don't know about. when I was using zigbee2Mqtt I don't have this issue. this issue only present after I switch to HASS Yellow and ZHA

@csolivan-ct
Copy link

Same Issue here had been working for years. What is the workaround here?

@rjenx
Copy link

rjenx commented Aug 31, 2023

Similar issue, lots of network failure messages nwk invalid request, mix of device manufacturers as they are all lights on this network.

Moved my WIFI to avoid overlap, WIFI 1 and 6, ZHA is currently on Band 25. Which is showing only 20% utilisation.

"energy_scan": {
  "11": 32.94117647058823,
  "12": 49.411764705882355,
  "13": 61.1764705882353,
  "14": 32.94117647058823,
  "15": 49.411764705882355,
  "16": 12.941176470588236,
  "17": 28.235294117647058,
  "18": 27.058823529411764,
  "19": 10.588235294117647,
  "20": 16.470588235294116,
  "21": 15.294117647058824,
  "22": 25.88235294117647,
  "23": 17.647058823529413,
  "24": 11.764705882352942,
  "25": 20.0,
  "26": 45.88235294117647

Removed coordinator (Slae.sh CC2652R) for an hour then reconnected.

Errors started re-appearing. Example below:

Logger: homeassistant
Source: components/zha/core/cluster_handlers/__init__.py:75
First occurred: 31 August 2023 at 23:00:04 (25 occurrences)
Last logged: 11:06:08

Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 64, in wrapper
    return await RETRYABLE_REQUEST_DECORATOR(func)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/util.py", line 132, in retry
    return await func()
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/zcl/__init__.py", line 375, in request
    return await self._endpoint.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/endpoint.py", line 253, in request
    return await self.device.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/device.py", line 293, in request
    await self._application.request(
  File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 824, in request
    await self.send_packet(
  File "/usr/local/lib/python3.11/site-packages/zigpy_znp/zigbee/application.py", line 1192, in send_packet
    raise DeliveryError(
zigpy.exceptions.DeliveryError: Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1199, in async_request_call
    return await coro
           ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 942, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 591, in async_handle_light_off_service
    await light.async_turn_off(**filter_turn_off_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/zha/light.py", line 471, in async_turn_off
    result = await self._on_off_cluster_handler.off()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/__init__.py", line 75, in wrapper
    raise HomeAssistantError(message) from exc
homeassistant.exceptions.HomeAssistantError: Failed to send request: Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>

Anecdotal - First noticed issues when automations starting failing.

@mathiasolofsson87
Copy link
Author

some new stuff from my logs. it fails to start zigpy.application. and then says the <EmberStatus.NETWORK_BUSY: 161>. So it seems like since 2023.7 zigbee/ZHA is polling devices too often? and that causes a flooding of the zigbee mesh if there are a lot of devices? I have 106 zigbee devices, but it was all running super smooth before 2023.7, now it's at the point where I need to restart HA every day due to ZHA not being able to send commands to devices, due to (my guess) the zigbee mesh network being overloaded from polling or other checks?

Here's an updated log excerpt from yesterday when I had issues where on my phone in the HA app it said Application Controller Not running instead of the usual EmberStatus.DELIVERY_FAILED 102 notificiation at the bottom of the screen when trying to turn on/off a light.

Attaching a full log file here, along with some excerpts below.
ZHAlog.txt

Starts off with this:

2023-09-06 19:31:51.640 ERROR (MainThread) [zigpy.application] Couldn't start application
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 194, in startup
await self.initialize(auto_form=auto_form)
File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 151, in initialize
await self.permit(0)
File "/usr/local/lib/python3.11/site-packages/bellows/zigbee/application.py", line 838, in permit
await super().permit(time_s, node)
File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 1136, in permit
await zigpy.zdo.broadcast(
File "/usr/local/lib/python3.11/site-packages/zigpy/device.py", line 519, in broadcast
return await app.broadcast(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 912, in broadcast
await self.send_packet(
File "/usr/local/lib/python3.11/site-packages/bellows/zigbee/application.py", line 812, in send_packet
raise zigpy.exceptions.DeliveryError(
zigpy.exceptions.DeliveryError: Failed to enqueue message after 3 attempts: <EmberStatus.NETWORK_BUSY: 161>
2023-09-06 19:31:51.659 WARNING (MainThread) [zigpy.appdb] Discarding _save_device_relays event
2023-09-06 19:31:51.671 WARNING (MainThread) [zigpy.appdb] Discarding _save_device_relays event
2023-09-06 19:31:51.725 WARNING (MainThread) [zigpy.appdb] Discarding _save_device_relays event
2023-09-06 19:31:51.737 WARNING (MainThread) [zigpy.appdb] Discarding _save_device_last_seen event
2023-09-06 19:31:51.759 WARNING (MainThread) [homeassistant.components.zha.core.gateway] Couldn't start EZSP = Silicon Labs EmberZNet protocol: Elelabs, HUSBZB-1, Telegesis coordinator (attempt 1 of 3)
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/zha/core/gateway.py", line 205, in async_initialize
self.application_controller = await app_controller_cls.new(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 219, in new
await app.startup(auto_form=auto_form)
File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 194, in startup
await self.initialize(auto_form=auto_form)
File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 151, in initialize
await self.permit(0)
File "/usr/local/lib/python3.11/site-packages/bellows/zigbee/application.py", line 838, in permit
await super().permit(time_s, node)
File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 1136, in permit
await zigpy.zdo.broadcast(
File "/usr/local/lib/python3.11/site-packages/zigpy/device.py", line 519, in broadcast
return await app.broadcast(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/zigpy/application.py", line 912, in broadcast
await self.send_packet(
File "/usr/local/lib/python3.11/site-packages/bellows/zigbee/application.py", line 812, in send_packet
raise zigpy.exceptions.DeliveryError(
zigpy.exceptions.DeliveryError: Failed to enqueue message after 3 attempts: <EmberStatus.NETWORK_BUSY: 161>
2023-09-06 19:31:54.803 WARNING (MainThread) [aiohttp.websocket] Client protocols ['http-only', 'chat'] don’t overlap server-known ones ()
2023-09-06 19:31:55.848 WARNING (MainThread) [homeassistant.components.sensor] Setup of sensor platform nordpool is taking over 10 seconds.
2023-09-06 19:32:02.641 WARNING (MainThread) [zigpy.application] Zigbee channel 20 utilization is 80.38%!
2023-09-06 19:32:02.644 WARNING (MainThread) [zigpy.application] If you are having problems joining new devices, are missing sensor updates, or have issues keeping devices joined, ensure your coordinator is away from interference sources such as USB 3.0 devices, SSDs, WiFi routers, etc.
2023-09-06 19:32:02.645 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 0 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.BasicClusterHandler'>
2023-09-06 19:32:02.645 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 6 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.OnOffClusterHandler'>
2023-09-06 19:32:02.645 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 10 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.Time'>
2023-09-06 19:32:02.645 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 25 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.Ota'>
2023-09-06 19:32:02.645 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 1281 class: <class 'homeassistant.components.zha.core.cluster_handlers.security.IasAce'>
2023-09-06 19:32:02.645 DEBUG (MainThread) [homeassistant.components.zha.core.gateway] [0x0000](Silicon Labs EZSP) restored as 'available', last seen: 0:00:05 ago, consider_unavailable_time: 7200 seconds
2023-09-06 19:32:02.645 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 0 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.BasicClusterHandler'>
2023-09-06 19:32:02.645 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 3 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.Identify'>
2023-09-06 19:32:02.645 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 4 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.Groups'>
2023-09-06 19:32:02.645 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 5 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.Scenes'>
2023-09-06 19:32:02.645 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 6 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.OnOffClusterHandler'>
2023-09-06 19:32:02.645 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 8 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.LevelControlClusterHandler'>
2023-09-06 19:32:02.646 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 4096 class: <class 'homeassistant.components.zha.core.cluster_handlers.lightlink.LightLink'>
2023-09-06 19:32:02.646 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 768 class: <class 'homeassistant.components.zha.core.cluster_handlers.lighting.ColorClusterHandler'>
2023-09-06 19:32:02.646 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 64513 class: <class 'homeassistant.components.zha.core.cluster_handlers.ClusterHandler'>
2023-09-06 19:32:02.646 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] Discovering entities for endpoint: 00:17:88:01:09:ee:ef:e8-11
2023-09-06 19:32:02.646 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'button' component -> 'ZHAIdentifyButton' using ['identify']
2023-09-06 19:32:02.646 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'sensor' component -> 'RSSISensor' using ['basic']
2023-09-06 19:32:02.646 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'sensor' component -> 'LQISensor' using ['basic']
2023-09-06 19:32:02.647 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'OnOffTransitionTimeConfigurationEntity' using ['level']
2023-09-06 19:32:02.647 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'OnLevelConfigurationEntity' using ['level']
2023-09-06 19:32:02.647 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'OnTransitionTimeConfigurationEntity' using ['level']
2023-09-06 19:32:02.647 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'OffTransitionTimeConfigurationEntity' using ['level']
2023-09-06 19:32:02.647 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'DefaultMoveRateConfigurationEntity' using ['level']
2023-09-06 19:32:02.647 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'StartUpCurrentLevelConfigurationEntity' using ['level']
2023-09-06 19:32:02.647 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'StartUpColorTemperatureConfigurationEntity' using ['light_color']
2023-09-06 19:32:02.647 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'select' component -> 'ZHAStartupOnOffSelectEntity' using ['on_off']
2023-09-06 19:32:02.647 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] Discovering entities for endpoint: 00:17:88:01:09:ee:ef:e8-242
2023-09-06 19:32:02.647 DEBUG (MainThread) [homeassistant.components.zha.core.gateway] [0x8E16](Philips 1743030P7) restored as 'available', last seen: 0:00:58 ago, consider_unavailable_time: 7200 seconds
2023-09-06 19:32:02.647 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 0 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.BasicClusterHandler'>
2023-09-06 19:32:02.647 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 3 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.Identify'>
2023-09-06 19:32:02.647 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 4 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.Groups'>
2023-09-06 19:32:02.647 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 5 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.Scenes'>
2023-09-06 19:32:02.647 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 6 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.OnOffClusterHandler'>
2023-09-06 19:32:02.647 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 8 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.LevelControlClusterHandler'>
2023-09-06 19:32:02.647 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 4096 class: <class 'homeassistant.components.zha.core.cluster_handlers.lightlink.LightLink'>
2023-09-06 19:32:02.647 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 768 class: <class 'homeassistant.components.zha.core.cluster_handlers.lighting.ColorClusterHandler'>
2023-09-06 19:32:02.647 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] Discovering entities for endpoint: 00:17:88:01:0c:68:81:31-11
2023-09-06 19:32:02.648 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'button' component -> 'ZHAIdentifyButton' using ['identify']
2023-09-06 19:32:02.648 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'sensor' component -> 'RSSISensor' using ['basic']
2023-09-06 19:32:02.648 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'sensor' component -> 'LQISensor' using ['basic']
2023-09-06 19:32:02.648 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'OnOffTransitionTimeConfigurationEntity' using ['level']
2023-09-06 19:32:02.648 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'OnLevelConfigurationEntity' using ['level']
2023-09-06 19:32:02.648 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'OnTransitionTimeConfigurationEntity' using ['level']
2023-09-06 19:32:02.648 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'OffTransitionTimeConfigurationEntity' using ['level']
2023-09-06 19:32:02.648 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'DefaultMoveRateConfigurationEntity' using ['level']
2023-09-06 19:32:02.648 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'StartUpCurrentLevelConfigurationEntity' using ['level']
2023-09-06 19:32:02.648 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'StartUpColorTemperatureConfigurationEntity' using ['light_color']
2023-09-06 19:32:02.648 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'select' component -> 'ZHAStartupOnOffSelectEntity' using ['on_off']
2023-09-06 19:32:02.648 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] Discovering entities for endpoint: 00:17:88:01:0c:68:81:31-242
2023-09-06 19:32:02.649 DEBUG (MainThread) [homeassistant.components.zha.core.gateway] [0x8264](Philips 915005996701) restored as 'available', last seen: 0:00:13 ago, consider_unavailable_time: 7200 seconds
2023-09-06 19:32:02.649 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 0 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.BasicClusterHandler'>
2023-09-06 19:32:02.649 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 3 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.Identify'>
2023-09-06 19:32:02.649 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 4 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.Groups'>
2023-09-06 19:32:02.649 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 5 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.Scenes'>
2023-09-06 19:32:02.649 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 6 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.OnOffClusterHandler'>
2023-09-06 19:32:02.649 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 8 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.LevelControlClusterHandler'>
2023-09-06 19:32:02.649 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 4096 class: <class 'homeassistant.components.zha.core.cluster_handlers.lightlink.LightLink'>
2023-09-06 19:32:02.649 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 768 class: <class 'homeassistant.components.zha.core.cluster_handlers.lighting.ColorClusterHandler'>
2023-09-06 19:32:02.649 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 64513 class: <class 'homeassistant.components.zha.core.cluster_handlers.ClusterHandler'>
2023-09-06 19:32:02.649 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] Discovering entities for endpoint: 00:17:88:01:09:ea:a7:78-11
2023-09-06 19:32:02.649 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'button' component -> 'ZHAIdentifyButton' using ['identify']
2023-09-06 19:32:02.649 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'sensor' component -> 'RSSISensor' using ['basic']
2023-09-06 19:32:02.649 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'sensor' component -> 'LQISensor' using ['basic']
2023-09-06 19:32:02.650 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'OnOffTransitionTimeConfigurationEntity' using ['level']
2023-09-06 19:32:02.650 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'OnLevelConfigurationEntity' using ['level']
2023-09-06 19:32:02.650 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'OnTransitionTimeConfigurationEntity' using ['level']
2023-09-06 19:32:02.650 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'OffTransitionTimeConfigurationEntity' using ['level']
2023-09-06 19:32:02.650 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'DefaultMoveRateConfigurationEntity' using ['level']
2023-09-06 19:32:02.650 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'StartUpCurrentLevelConfigurationEntity' using ['level']
2023-09-06 19:32:02.650 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'StartUpColorTemperatureConfigurationEntity' using ['light_color']
2023-09-06 19:32:02.650 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'select' component -> 'ZHAStartupOnOffSelectEntity' using ['on_off']
2023-09-06 19:32:02.650 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] Discovering entities for endpoint: 00:17:88:01:09:ea:a7:78-242
2023-09-06 19:32:02.651 DEBUG (MainThread) [homeassistant.components.zha.core.gateway] [0x0055](Philips 1745630P7) restored as 'available', last seen: 0:01:00 ago, consider_unavailable_time: 7200 seconds
2023-09-06 19:32:02.651 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 0 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.BasicClusterHandler'>
2023-09-06 19:32:02.651 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 3 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.Identify'>
2023-09-06 19:32:02.651 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 4 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.Groups'>
2023-09-06 19:32:02.651 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 5 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.Scenes'>
2023-09-06 19:32:02.651 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 6 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.OnOffClusterHandler'>
2023-09-06 19:32:02.651 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 8 class: <class 'homeassistant.components.zha.core.cluster_handlers.general.LevelControlClusterHandler'>
2023-09-06 19:32:02.651 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 4096 class: <class 'homeassistant.components.zha.core.cluster_handlers.lightlink.LightLink'>
2023-09-06 19:32:02.651 INFO (MainThread) [homeassistant.components.zha.core.endpoint] Creating cluster handler for cluster id: 768 class: <class 'homeassistant.components.zha.core.cluster_handlers.lighting.ColorClusterHandler'>
2023-09-06 19:32:02.651 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] Discovering entities for endpoint: 00:17:88:01:09:87:80:b5-11
2023-09-06 19:32:02.651 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'button' component -> 'ZHAIdentifyButton' using ['identify']
2023-09-06 19:32:02.651 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'sensor' component -> 'RSSISensor' using ['basic']
2023-09-06 19:32:02.651 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'sensor' component -> 'LQISensor' using ['basic']
2023-09-06 19:32:02.651 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'OnOffTransitionTimeConfigurationEntity' using ['level']
2023-09-06 19:32:02.651 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'OnLevelConfigurationEntity' using ['level']
2023-09-06 19:32:02.651 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'OnTransitionTimeConfigurationEntity' using ['level']
2023-09-06 19:32:02.651 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'OffTransitionTimeConfigurationEntity' using ['level']
2023-09-06 19:32:02.651 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'DefaultMoveRateConfigurationEntity' using ['level']
2023-09-06 19:32:02.651 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'StartUpCurrentLevelConfigurationEntity' using ['level']
2023-09-06 19:32:02.651 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'number' component -> 'StartUpColorTemperatureConfigurationEntity' using ['light_color']
2023-09-06 19:32:02.651 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'select' component -> 'ZHAStartupOnOffSelectEntity' using ['on_off']
2023-09-06 19:32:02.651 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] Discovering entities for endpoint: 00:17:88:01:09:87:80:b5-242

Then it keeps doing these clusterhandlers for a long time.

Then it does this over and over:

2023-09-06 19:32:03.053 DEBUG (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0x29ED:1:0x0008]: initializing cached cluster handler attributes: ['on_off_transition_time', 'on_level', 'on_transition_time', 'off_transition_time', 'default_move_rate', 'start_up_current_level']
2023-09-06 19:32:03.053 DEBUG (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0x29ED:1:0x0008]: Reading attributes in chunks: ['on_off_transition_time', 'on_level', 'on_transition_time', 'off_transition_time', 'default_move_rate']
2023-09-06 19:32:03.053 DEBUG (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0x29ED:1:0x0008]: Reading attributes in chunks: ['start_up_current_level']
2023-09-06 19:32:03.053 DEBUG (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0x29ED:1:0x0008]: initializing uncached cluster handler attributes: ['current_level'] - from cache[False]
2023-09-06 19:32:03.053 DEBUG (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0x29ED:1:0x0008]: Reading attributes in chunks: ['current_level']
2023-09-06 19:32:03.053 DEBUG (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0x29ED:1:0x0003]: initializing cluster handler: from_cache: False
2023-09-06 19:32:03.053 DEBUG (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0x29ED:1:0x0003]: finished cluster handler initialization
2023-09-06 19:32:03.053 DEBUG (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0x29ED:1:0x0000]: initializing cluster handler: from_cache: False
2023-09-06 19:32:03.053 DEBUG (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0x29ED:1:0x0000]: finished cluster handler initialization
2023-09-06 19:32:03.053 DEBUG (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0x29ED:1:0x1000]: initializing cluster handler: from_cache: False
2023-09-06 19:32:03.053 DEBUG (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0x29ED:1:0x1000]: finished cluster handler initialization
2023-09-06 19:32:03.053 DEBUG (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0x29ED:1:0x0019]: initializing cluster handler: from_cache: False
2023-09-06 19:32:03.053 DEBUG (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0x29ED:1:0x0019]: finished cluster handler initialization
2023-09-06 19:32:03.053 DEBUG (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0x8066:1:0x0006]: initializing cluster handler: from_cache: False
2023-09-06 19:32:03.053 DEBUG (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0x8066:1:0x0006]: initializing cached cluster handler attributes: ['start_up_on_off']
2023-09-06 19:32:03.053 DEBUG (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0x8066:1:0x0006]: Reading attributes in chunks: ['start_up_on_off']
2023-09-06 19:32:03.053 DEBUG (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0x8066:1:0x0006]: initializing uncached cluster handler attributes: ['on_off'] - from cache[False]
2023-09-06 19:32:03.053 DEBUG (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0x8066:1:0x0006]: Reading attributes in chunks: ['on_off']
2023-09-06 19:32:03.053 DEBUG (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0x8066:1:0x0300]: initializing cluster handler: from_cache: False
2023-09-06 19:32:03.053 DEBUG (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0x8066:1:0x0300]: initializing cached cluster handler attributes: ['color_temp_physical_min', 'color_temp_physical_max', 'color_capabilities', 'start_up_color_temperature', 'options']
2023-09-06 19:32:03.053 DEBUG (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0x8066:1:0x0300]: Reading attributes in chunks: ['color_temp_physical_min', 'color_temp_physical_max', 'color_capabilities', 'start_up_color_temperature', 'options']

Then in the end I get:

cluster: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>
2023-09-06 19:32:07.287 DEBUG (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0xE831:1:0x0008]: async_initialize: retryable request #1 failed: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>. Retrying in 0.8s
2023-09-06 19:32:07.319 DEBUG (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0xE42A:1:0x0006]: failed to get attributes '['on_off']' on 'on_off' cluster: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>
2023-09-06 19:32:07.319 DEBUG (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0xE42A:1:0x0006]: async_initialize: retryable request #1 failed: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>. Retrying in 1.2s
2023-09-06 19:32:07.338 ERROR (MainThread) [zigpy.zcl] [0xE762:1:0x0020] AssertionError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/zigpy/device.py", line 292, in request
with self._pending.new(sequence) as req:
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/zigpy/util.py", line 291, in new
raise ControllerException(f"duplicate {sequence} TSN") from AssertionError
zigpy.exceptions.ControllerException: duplicate 17 TSN

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/general.py", line 538, in check_in_response
await self.checkin_response(True, self.CHECKIN_FAST_POLL_TIMEOUT, tsn=tsn)
File "/usr/src/homeassistant/homeassistant/components/zha/core/cluster_handlers/init.py", line 75, in wrapper
raise HomeAssistantError(message) from exc
homeassistant.exceptions.HomeAssistantError: Failed to send request: duplicate 17 TSN

@somerandomuser1
Copy link

my ZHA just broke again. tried repairing, nothing, and i mean NOTHING, will pair, not my Endpoints, not my Router devices, nothing. I tried rolling back to the last stable build which usually fixes this for me, but its still busted. ETA on a fix for this nonsense plz?

@somerandomuser1
Copy link

After a bit of testing, incase anyone else has this issue, the ONLY thing that seems to fix it for me is to move my Coordinator to a different USB port, literally any port, as long as its a different one from the one that it was previously in. After doing that I can now get pairing to work again.

@teijosantala
Copy link

I also have the same issue.
I have reverted back to 2023.7.3 where ZHA has no issues for me.
My coordinator is SonOFF ZBDongle-E 3.0 USB Dongle Plus (if that make any difference).

@lacojim
Copy link

lacojim commented Sep 13, 2023

This is a newly acquired bug for me too. I have been fighting this problem for over a month now. I need to figure out how to revert and keep it from auto-updating. Ugh

@BenPru
Copy link

BenPru commented Sep 13, 2023

I have reverted back to 2023.7.3

I have this problem since HA 2023.8.4
2023.8.3 works fine

@dumpfheimer
Copy link

I might be ve completely off here but I spent quite some time with similar errors today (delivery errors, extremely unresponsive network) and after hours of debugging and checking interference and what else, it seems like..

I found a misbehaving device

(Philips Hue light strip). As soon as I unplugged it for a few minutes everything instantly got snappy again and worked perfectly fine.

Even though it seems like your issues seem to be related to HA versions, I thought I might just put this here, maybe someone is having the same issue and is spared a few very frustrating hours.

@mathiasolofsson87
Copy link
Author

I did find a light (ikea trådfri driver) that looked like it was fine in the interface, but in reality was not responding. (Closet we rarely open) so tried reconnecting it so its working again and will see if things improve

@rjenx
Copy link

rjenx commented Sep 14, 2023

I have downgraded from 2023.8 to 2023.7.3 as I note that in 2023.8 there is a fix #97539 which purportedly resolves an issue with service light.turn_on. Its this service failing in automations that alerted me to the original problem. I will report back any observations.

@rjenx
Copy link

rjenx commented Sep 14, 2023

I might be ve completely off here but I spent quite some time with similar errors today (delivery errors, extremely unresponsive network) and after hours of debugging and checking interference and what else, it seems like..

I found a misbehaving device

(Philips Hue light strip). As soon as I unplugged it for a few minutes everything instantly got snappy again and worked perfectly fine.

Even though it seems like your issues seem to be related to HA versions, I thought I might just put this here, maybe someone is having the same issue and is spared a few very frustrating hours.

How did you determine the defective device?

@rjenx
Copy link

rjenx commented Sep 14, 2023

I have downgraded from 2023.8 to 2023.7.3 as I note that in 2023.8 there is a fix #97539 which purportedly resolves an issue with service light.turn_on. Its this service failing in automations that alerted me to the original problem. I will report back any observations.

A little better stability but still some devices failing

@lacojim
Copy link

lacojim commented Sep 15, 2023

Reverting back to 2023.7.2 fixes the issue for me finally.

@somerandomuser1
Copy link

well, once again, my endpoints and a few routers became unavailable and I could not get them to pair. This time I decided to pull the trigger and downgrade to 7.1. Doing so enabled me to Pair my devices instantly with no issues, no changing USB ports, no wiping out the network and starting from scratch, it just pairs right away the first try. Will report back if I have the issue anymore, hopefully this fixes it for me.

Incase anyone else is wanting to downgrade to an earlier version to solve this issue, here's how.

In the WebGUI go to settings>addons> install the "Terminal and SSH" addon. Wait for it to finish installing, theres an option in the addon to enable viewing it in the sidebar, enable that, and Start the addon. Wait a few minutes, then on the left hand side you'll see "Terminal" appear. Click it, once in the Terminal, this is the command to type in: ha core update --version=2023.7.1
You can use whichever version you want here as long as its a valid version number. I used 7.1. After its all done you'll need to Reboot the machine running HA. You can then go back into Terminal and it will show the current HA Core version number. If it shows the version you wanted to downgrade to, success. If not, you'll need to try again/google around for a solution.

Hope this helps.

@MattWestb
Copy link
Contributor

The Thread integration with Zigbeed is no problem if its enabled or not then one of my 3 test system is running with over 30 devices most IKEA controllers also blinds and have Matter server on the same install and the thread and Zigbee is rock solid.

@ghost
Copy link

ghost commented Oct 2, 2023

@ekalle-swe Can you upload diagnostics for the integration again?

I've added the lines with the ZHA logging to the config file and restarted HA. God is a clown so naturally it started to behave well again and I can't even provoke the problem. Well, sooner or later it will and then I will post the file for you.

@ghost
Copy link

ghost commented Oct 4, 2023

@puddly

Now the ZHA has failed. Nothing Zigbee is working but all non Zigbee (wifi and Telldus) are still ok, so it's not HA that has crashed.
Have both the home-assistant.log file and ZHA diagnostics file for you.
Strange thing. Wasn't even possible to download the ZHA diagnostic file. Download failed with "file not accessible on web site" until I did a restart. Then it worked. Don't know if the diagnostic file has any value after a restart.
Do you want them both?
Home-assistant.log is rather large (over 600MB, 1,9Mlines). Do you want the whole ting or is a tail -1000 enough?

@puddly
Copy link
Contributor

puddly commented Oct 4, 2023

ZIP them both and email them to me please, if they're too big to post here.

@mathiasolofsson87
Copy link
Author

Shouldve kept mybstupid mouth shut. After updating from 2023.9.2 to 2023.9.3 I get the same error 102 every day and all zigbee devices stop responding until restart of the system.

@Californian
Copy link

I disabled thread, openthreadborderrouter, and matter and enabled them after boot and now it seems to work more reliably.

@ghost
Copy link

ghost commented Oct 11, 2023

Problem accelerating again.
After I moved Zigbee channel, it could be days between Zigbee krashes. Now I'm back to reloading Zha several times every day. Updates to 23.10.0 and 23.10.1 did not bring anything to solve this.
New tendency is that several times only parts of the network dies and the scary thing is that it's not consistently the same devices that get in trouble or devices connected to the same repeater. It's completely random. Seams like it's more a question of devices that was in a surtain state when a problem occurred. The rest keeps on working.
This problem takes up to much time an energy from me and frustration for the family. Thinking of moving away from ZHA but my gut instinct tells me that it's unproductive to flee from one known problem to a completely unknown territory that may well bring you new and more problems.

@mathiasolofsson87
Copy link
Author

I located the issue to my problem happening again. My wife had moved the Zigbee SKyConnect stick on top of the HA NUC :) moved it away again and now it is stable once more. My own fault since I never informed her it cant lie on top of the NUC due to interference. So now all seems to be ok again since at least 4 days.

@MattWestb
Copy link
Contributor

Sirlabs have (yesterday evening CET) released one update with some bug fixes that can fitting in this case but we will see if the devs will cooking one updated firmware that fixing some of the network steering problems.

@hcross13
Copy link
Contributor

hcross13 commented Oct 12, 2023

I too have had the same issues, where Zigbee devices become unavailable/offline in Home Assistant.
I was using a ConBee II, on a Raspberry Pi 4, running HASS OS with 42 Zigbee end devices and 3 Zigbee repeaters spread around the house and the configuration was working flawlessly for over a year.
I updated the ConBee II firmware and didn't really see any improvement so switched to the SkyConnect device to rule out the hardware, but it too encountered the same issues with the ZHA integration.

I deleted the ZHA integration and started again from scratch, using channel 25 which had a consistent energy_scan of 1-2%, re-pairing every device manually, but what I noticed was that as soon as I paired around 8-10 devices, the warning message "Zigbee channel 25 utilization is 89.67%!" would be between 85-95% and then devices would start to stuggle pairing and going offline.

I fought this for a few days and tried 3 separate channels, using different USB extension cables (0.5m, 1m, 2m, 3m ,5m, 15m) with no difference in behaviour before I deleted the ZHA integration and set up the Zigbee2MQTT integration/addon which is working perfectly with the short 0.5m USB extension that comes with the SkyConnect (possibly even more responsive than ZHA was and returns more details about the Zigbee devices, with a better map showing the device connections), so I can definitely recommend Zigbee2MQTT for those considering another option (you can always take a backup an migrate back to ZHA once the issue is resolved)

I wanted to share my information as there certainly appears to have been an issue introduced in the ZHA integration, which has gotten worse over time and all debugging and reconfiguration cannot resolve the issue.
I'm unlikely to come back to the ZHA integration in a hurry now that I've spent a further 12 hours switching to the Zigbee2MQTT integration/addon and recreated all of my entities and updated all automations, etc as it works brilliantly with more information, but I may restore a backup and try it again in future to see if thngs have improved.

@MattWestb
Copy link
Contributor

@hcross13 45 devices and only 3 routers can working well if you have good Zigbee 3 routers and the coordinator is doing all things OK if the routers have good RF coverage (that is unlikely you is having) as long you is not getting other if signals around that is blocking the radio channel and the network cant not rerouting they your network is on star network on the router side an not one self healing mesh network.
Read the good papers out in the internet for understanding the problematic.

@ghost
Copy link

ghost commented Oct 12, 2023

Sorry but I seriously doubt that you can build a 48 device Zigbee network and only have 3 router, as nearly everything not running on battery is a router.
Don't you have wall plugs, smart lights, smart led strips like the rest of us?

@hcross13
Copy link
Contributor

@ekalle-swe I'm using powered Tuya Zigbee light switches, but they are 2 wire switches (no neutral wire), which do not function as routers, so I need the 3 repeaters to ensure a smooth and reliable network coverage within the house.

As I mentioned, everything was working fine for over a year with the exact same hardware in ZHA, then I started getting devices going offline, so I included the details for reference and @MattWestb I have read through the articles to try and reduce interference since encountering the recent problems, but I suspect that there's something wrong in the ZHA integration, that may only manifest under some scenarios.

After recently migrating to Zigbee2MQTT my Zigbee network and automations are stable, reliable and fast, which I hadn't had with ZHA for a few months.
This is my personal experience, I'm unable to use ZHA reliably with more than 10 devices, but everything works perfectly with Zigbee2MQTT using the exact same hardware.

Here's a screenshot of the map showing the Zigbee network
image

@MattWestb
Copy link
Contributor

Your network is working in star mode the routers is only talking with the coordinator and not using the neighbors = no redundancy / rerouting possible.
And if looking on the router LQI for all links they is asymmetrical = shall being symmetrical for working best (sending and reviving LQI is nearly the same).
Also your "routers" is worse ones so you need better routers and more of them for getting one good working mesh network that can rerouting then getting interference that is not can d in your case.

I have my production network overlapping 100% with my WiFi (with over 150 WiFi devices most ESPHome) and its working good but not perfect then i have one backbone of IKEA plugs that can going thru also then having very strong signals from other systems.

So having over 10 children connected to one good router (the chip (MG21) is OK but the antenna is very bad) you cant getting it working well its only time you is getting more problems also with Z2M and DEConz then its not the host system that is bad its your network is wrong build and is better using only WiFi devices if you like building like that with all its bad things like more single point of failure.

@MattWestb
Copy link
Contributor

In ZHA you shall putting this in your config after have adding at least one router for getting your network working better:

    source_routing: false
    ezsp_config:
      CONFIG_MAX_END_DEVICE_CHILDREN: 0

Its making the coordinator not accepting children so its must using one other router and also disabling source routing then its only making problems in your configuration.
(also working well with all RCP coordinators in ZHA than i have one of the test system running with over 30 test devices most IKEA controller)

But in the end one mesh network needs good routers for working well !!!

@hcross13
Copy link
Contributor

@MattWestb thanks for the helpful information, I may look to change some hardware in the near future and give ZHA another go.

I still don't understand why it was working in ZHA for over a year without issues and why it still works in Z2M now, but has started getting issues with ZHA in recent months with no changes to hardware, only software changes/updates to ZHA/HA?

@ghost
Copy link

ghost commented Oct 12, 2023

Did you start your Zigbee voyage with wall switches and have the rest on wifi/433/...?

@MattWestb
Copy link
Contributor

You is not knowing what radio interference have changed around you network and then its not one valid statement you have going over one limits and getting more and more problem.
Also one end device is normally using 1/10 power of one router so you shall having routers around all end device for getting them working OK.

Some fundamental readings: https://www.silabs.com/documents/public/user-guides/ug103-02-fundamentals-zigbee.pdf

@puddly
Copy link
Contributor

puddly commented Oct 12, 2023

I still don't understand why it was working in ZHA for over a year without issues and why it still works in Z2M now, but has started getting issues with ZHA in recent months with no changes to hardware, only software changes/updates to ZHA/HA?

I don't doubt that your network works now. I want to figure out why, as I have no direction to look without more information about what is being done differently, if anything.

Can you describe:

  • what channels you tried with ZHA and what channel Z2M is currently using
  • what devices are disconnecting
  • what routers you have on your network
  • did you change the order in which you joined devices after rejoining them?

@ghost
Copy link

ghost commented Oct 25, 2023

Been running flawlessly since 23.10.5
Not one single incident.
Feels great!

@ghost
Copy link

ghost commented Nov 14, 2023

Here we go again!
With 23.11.2 the hangings have returned!
I was so happy. Been weeks since last problem but now I'm back to reloading ZHA several times every day.
Most of the time some devices are still working but the strange thing is that it also switches som devices on that hasn't been touched during the problem.
For instance, 1 individual light in a light group is being turned on spontaneously even if member devices are hidden and that group has not been touched since the day before, like the spotlight group in our basement entrance.

@dumpfheimer
Copy link

I have been having issues again too. This time, though, it was a bunch of NETWORK_BUSY errors (hope I remember the wording correctly). Not a single device was reachable, nor did groups seem to work. The error could be resolved by restarting HA.

@ghost
Copy link

ghost commented Dec 12, 2023

Here we go again.
After a relatively stable period, things are getting chaotic again. This morning I've had 4 of these ZHA hangings. Helps temporarily to reload ZHA, but not for long.
Took a look at the network map to see if things been jumping around again and just as last time when it was rough, some devices are not playing by the book.
In both cases it's Sonoff Temp/humid sensor (eWeLink H01) that is acting strange.
As you can see in the pictures, one is connected directly to the Skyconnect again (moved it before) and the other seams to have double connections.
I know that the coordinator doesn't like to have EndDevices connected, so that could be one problem. EndDevices with multiple connections could also confuse the network to think it's a router, so that could also be a problem.(speculations)

@puddly, maybe a clue.

Edit:

After again removing those two temp sensors and adding them under a router, the Zigbee network took a big sigh of relief and has worked flawlessly since.
The sensor with double attachments has still two attachments so that was probably not the problem.
The sensor that was connected to the coordinator is now away from the coordinator and connected to a router. That made a huge difference!
Thing is, when I look back, an EndDevice had jumped over to the coordinator last time I had huge problems as well.
Why does EndDevices suddenly jump over to the coordinator? Why does it only include my sonoff temp sensors and once an Aqara inwall double rocket (no neutral)? Why does the Zigbee network suffer so greatly because of one single EndDevice connected to the coordinator?
For stability, it would be far better to always deny EndDevices connection to the coordinator.

Skärmbild från 2023-12-12 11-52-26
Skärmbild från 2023-12-12 11-51-17

@ghost
Copy link

ghost commented Dec 17, 2023

Reload in 23.12.2 did not solve the problem.
The reload seams to miss some devices. Maybe timing issue. Maybe reload comes to quickly after an issue.
Anyway, some devices can still become unreachable but a manual reload fixes everything.
Will have to reactivate the scheduled reload service. (Would be great with a service that could reload one specific integration instead of just _all.)
As an example, these 2 devices where unreachable this morning but reload fixed it

Screenshot_20231217-080615
Screenshot_20231217-080447

@puddly
Copy link
Contributor

puddly commented Dec 17, 2023

This issue hasn't had any activity from the original author in a few months.

Please open a new issue if you have problems. The issue you're currently describing is not related to this one and is being tracked in other issues.

@puddly puddly closed this as completed Dec 17, 2023
@ghost
Copy link

ghost commented Dec 17, 2023

Most interesting!
What issue # is that?

@TomHejret
Copy link

What issue # is that?

It could be this one: #97662

@github-actions github-actions bot locked and limited conversation to collaborators Jan 25, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests