MQTT Exception raised while updating state of sensor BatteryStateAttribute/level' with payload: b'88'

[sca075]

The problem

Since beta 4 of 2024.6.0 this error is raising up randomly. It is understood that MQTT is trying to convert the payload data and as well want to return it as string or float, None (probably device wise network issue) was received probably raising this exception.

What version of Home Assistant Core has the issue?

2024.6.b04

What was the last working version of Home Assistant Core?

2024.5.4

What type of installation are you running?

Home Assistant OS

Integration causing the issue

MQTT

Link to integration documentation on our website

https://www.home-assistant.io/integrations/mqtt/

Diagnostics information

config_entry-mqtt-340dc203ad299d5bc055fdb889a578c5.json

Example YAML snippet

No response

Anything in the logs that might be useful for us?

Logger: homeassistant.components.mqtt.models
Source: components/mqtt/models.py:380
integration: MQTT (documentation, issues)
First occurred: 10:33:14 AM (1 occurrences)
Last logged: 10:33:14 AM

Exception raised while updating state of sensor.valetudo_glossyhardtofindnarwhal_battery_level, topic: 'valetudo/GlossyHardToFindNarwhal/BatteryStateAttribute/level' with payload: b'88'
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/sensor/__init__.py", line 658, in state
    numerical_value = int(value)
                      ^^^^^^^^^^
ValueError: invalid literal for int() with base 10: "b'88'"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/sensor/__init__.py", line 661, in state
    numerical_value = float(value)
                      ^^^^^^^^^^^^
ValueError: could not convert string to float: "b'88'"

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/mqtt/models.py", line 380, in process_write_state_requests
    entity.async_write_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1000, in async_write_ha_state
    self._async_write_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1126, in _async_write_ha_state
    state, attr, capabilities, shadowed_attr = self.__async_calculate_state()
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1061, in __async_calculate_state
    state = self._stringify_state(available)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1006, in _stringify_state
    if (state := self.state) is None:
                 ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/sensor/__init__.py", line 665, in state
    raise ValueError(
ValueError: Sensor sensor.valetudo_glossyhardtofindnarwhal_battery_level has device class 'battery', state class 'None' unit '%' and suggested precision 'None' thus indicating it has a numeric value; however, it has the non-numeric value: 'b'88'' (<class 'str'>)

### Additional information

According to #116437 By default HA try to decode incoming payloads as utf-8 on the supported entity platforms. The error comes from the vacuum "battery_state" then 'b'88'' decoded in utf-8 = 88 is that right?  

[home-assistant]

Hey there @emontnemery, @jbouwh, @bdraco, mind taking a look at this issue as it has been labeled with an integration (mqtt) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of mqtt can trigger bot actions by commenting:

• @home-assistant close Closes the issue.
• @home-assistant rename Awesome new title Renames the issue.
• @home-assistant reopen Reopen the issue.
• @home-assistant unassign mqtt Removes the current integration label and assignees on the issue, add the integration domain after the command.
• @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue.
• @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

---

mqtt documentation
mqtt source
_{^{(message by IssueLinks)}}

[jbouwh]

Is seems an issue with decoding the incoming payload. But also the error handling with it. To solve this as incident, the config for the sensor should be checked. Possible the encoding attribute is set to null in the config payload.
I cannot find the config of the failing sensor in the diagnostics data. Could you share the configuration payload of the sensor:
Should be published at a topic like homeassistant/sensor/xxxx/xxxx/config. You can use MQTT explorer to discover this.

[sca075]

@jbouwh I would provide the MQTT explorer details but I'm currently aboard.. and I can't collect it right now. Sorry for that.
You are correct it looks the vacuum sent a NULL value.
Is there any way you could handle this scenario somehow?
As soon I will be back home I will try to reproduce the issue, anyway it looks that when the value goes <100% (changing and docked no issue) the conversion failed wile "cleaning" hope this helps.

[jbouwh]

The linked PR will fix the error stack is dumped, but a warning will logged instead. It seems the encoding options is set to "", in which case the payload is not decoded as utf-8. This needs to be fixed in the integration/service or device that supplies the MQTT config payload.

[sca075]

Thanks a lot @jbouwh for taking care about this on a Sunday, really appreciated. Thanks also at the complete team :) amazing work guys. At the end of next week I will be back home and do some more test, also collect the MQTT payload from MQTT Explorer and will pass the logs of the vacuum to the authors of the vacuum firmware (that by the way is a Valetudo)

[jbouwh]

Feel free to share the configs later. The issue was closed automatically as a fix was merged. But note that this will not solve your issue unless the configuration for your entity is fixed.

[sca075]

Will do as soon I'm back home, thanks again.

[marvinwankersteen]

Is there a workaround to get around the problem before it is fixed in Valetudo (if I have understood it correctly)?

[sca075]

@jbouwh sorry for so long let you waiting, as requested, from MQTT explorer:
It looks according to what discussed the payload is configured correctly: homeassistant/sensor/GlossyHardToFindNarwhal/GlossyHardToFindNarwhal_sensor_battery_level/config

in the config of the sensor:

{"state_topic":"valetudo/GlossyHardToFindNarwhal/BatteryStateAttribute/level","icon":"mdi:battery","entity_category":"diagnostic","unit_of_measurement":"%","device_class":"battery","name":"Battery level","object_id":"valetudo_glossyhardtofindnarwhal_battery level","unique_id":"GlossyHardToFindNarwhal_sensor_battery_level","availability_topic":"valetudo/GlossyHardToFindNarwhal/$state","payload_available":"ready","payload_not_available":"lost","device":{"manufacturer":"Roborock","model":"S5","name":"Valetudo S5 GlossyHardToFindNarwhal","identifiers":["GlossyHardToFindNarwhal"],"sw_version":"Valetudo 2024.02.0","configuration_url":"http://valetudo-glossyhardtofindnarwhal.local"}}

The value of the sensor (is an Integer) according to MQTT explorer is an int but there is no Null recoded in valetudo/GlossyHardToFindNarwhal/BatteryStateAttribute/level

Hope this can help, please let me know if anything else is needed.

[marvinwankersteen]

I have recently updated to Home Assistant 2024.7.1 and also to Valetudo 2024.06.2, but the problem still persists.

home assistant log:

2024-07-07 11:57:33.388 DEBUG (MainThread) [homeassistant.components.mqtt.client] Received message on staubsauger/robo/hass/robo_vacuum/state (qos=0): b'{"state":"docked","battery_level":66,"fan_speed":"medium"}'
2024-07-07 11:57:33.388 DEBUG (MainThread) [homeassistant.components.mqtt.client] Received message on staubsauger/robo/BatteryStateAttribute/level (qos=0): b'66'
2024-07-07 11:57:33.389 WARNING (MainThread) [homeassistant.components.mqtt.sensor] Invalid undecoded state message 'b'66'' received from 'staubsauger/robo/BatteryStateAttribute/level'

mqtt sub homeassistant/sensor/robo/robo_sensor_battery_level/config:

{
  "state_topic": "staubsauger/robo/BatteryStateAttribute/level",
  "icon": "mdi:battery",
  "entity_category": "diagnostic",
  "unit_of_measurement": "%",
  "device_class": "battery",
  "name": "Battery level",
  "object_id": "valetudo_robo_battery level",
  "unique_id": "robo_sensor_battery_level",
  "availability_topic": "staubsauger/robo/$state",
  "payload_available": "ready",
  "payload_not_available": "lost",
  "device": {
    "manufacturer": "Roborock",
    "model": "V1",
    "name": "Valetudo V1 SharpAdoredCobra",
    "identifiers": [
      "robo"
    ],
    "sw_version": "Valetudo 2024.06.2",
    "configuration_url": "http://valetudo-sharpadoredcobra.local"
  }
}

mqtt sub staubsauger/robo/BatteryStateAttribute/level:

$ docker run -ti hivemq/mqtt-cli sub -h <mqttbroker> -u <user> -pw=<password> -t staubsauger/robo/BatteryStateAttribute/level -vJ

Command SubscribeCommand{mqttClientExecutor=com.hivemq.cli.mqtt.MqttClientExecutor@4149c063, subscribeClient=null, logToLogfile=false, connectOptions=ConnectOptions{version=null, host='<mqttbroker>', port=null, identifier='null', identifierPrefix='null', keepAlive=null, cleanStart=true, sessionExpiryInterval=null, connectUserProperties=null, useWebSocket=false, webSocketPath='null', willOptions=WillOptions{willTopic='null', willMessage=null, willQos=AT_MOST_ONCE, willRetain=false, willMessageExpiryInterval=null, willDelayInterval=null, willPayloadFormatIndicator=null, willContentType='null', willResponseTopic='null', willCorrelationData=null, willUserProperties=null}, connectRestrictionOptions=ConnectRestrictionOptions{receiveMaximum=null, sendMaximum=null, maximumPacketSize=null, sendMaximumPacketSize=null, topicAliasMaximum=null, sendTopicAliasMaximum=null, requestProblemInformation=true, requestResponseInformation=false}, authenticationOptions=AuthenticationOptions{user='<user>', password=java.nio.HeapByteBuffer[pos=0 lim=12 cap=12]}, tlsOptions=TlsOptions{useSsl=false, serverCertificates=null, serverCertificatesFromDir=null, cipherSuites=null, supportedTLSVersions=null, clientCertificates=null, clientPrivateKey=null}}, subscribeOptions=SubscribeOptions{topics=[staubsauger/robo/BatteryStateAttribute/level], qos=[EXACTLY_ONCE], userProperties=null, outputFile=null, printToSTDOUT=true, base64=false, jsonOutput=true, showTopics=false}, debugOptions=DebugOptions{isDebug=true, isVerbose=true}, helpOptions=HelpOptions{versionInfoRequested=false, usageHelpRequested=false}}
Setting value of 'version' to default value: MQTT_5_0
Setting value of 'port' to default value: 1883
Empty identifier will lead to using broker generated client identifier
Client 'UNKNOWN@<mqttbroker>' sending CONNECT
    MqttConnect{keepAlive=60, cleanStart=true, sessionExpiryInterval=0, simpleAuth=MqttSimpleAuth{username and password}}
Client 'UNKNOWN@<mqttbroker>' received CONNACK
    MqttConnAck{reasonCode=SUCCESS, sessionPresent=false, assignedClientIdentifier=auto-49F895E4-A7D9-B668-CCCC-04DC0054C628, restrictions=MqttConnAckRestrictions{receiveMaximum=20, maximumPacketSize=268435460, topicAliasMaximum=10, maximumQos=EXACTLY_ONCE, retainAvailable=true, wildcardSubscriptionAvailable=true, sharedSubscriptionAvailable=true, subscriptionIdentifiersAvailable=true}}
Client 'auto-49F895E4-A7D9-B668-CCCC-04DC0054C628@<mqttbroker>' sending SUBSCRIBE
    MqttSubscribe{subscriptions=[MqttSubscription{topicFilter=staubsauger/robo/BatteryStateAttribute/level, qos=EXACTLY_ONCE, noLocal=false, retainHandling=SEND, retainAsPublished=false}]}
Client 'auto-49F895E4-A7D9-B668-CCCC-04DC0054C628@<mqttbroker>' received SUBACK
    MqttSubAck{reasonCodes=[GRANTED_QOS_2], packetIdentifier=65526}
Client 'auto-49F895E4-A7D9-B668-CCCC-04DC0054C628@<mqttbroker>' received PUBLISH ('64')
    MqttPublish{topic=staubsauger/robo/BatteryStateAttribute/level, payload=2byte, qos=AT_LEAST_ONCE, retain=true}
{
  "topic": "staubsauger/robo/BatteryStateAttribute/level",
  "payload": 64,
  "qos": "AT_LEAST_ONCE",
  "receivedAt": "2024-07-07 09:55:38",
  "retain": true
}
Client 'auto-49F895E4-A7D9-B668-CCCC-04DC0054C628@<mqttbroker>' sending PUBACK
    MqttPubAck{reasonCode=SUCCESS, packetIdentifier=1}

[sca075]

I think we need to open an issue to Valetudo as per the configuration of the vacuums needs to be updated.

'''
Logger: homeassistant.components.mqtt.vacuum
Source: components/mqtt/vacuum.py:173
integration: MQTT (documentation, issues)
First occurred: July 10, 2024 at 10:20:30 PM (1 occurrences)
Last logged: July 10, 2024 at 10:20:30 PM

The schema option is deprecated for MQTT vacuum, but it was used in a discovery payload. Please contact the maintainer of the integration or service that supplies the config, and suggest to remove the option. Got {'object_id': 'valetudo_silenttepidstinkbug', 'schema': 'state', 'supported_features': ['battery', 'status', 'start', 'stop', 'pause', 'return_home', 'fan_speed', 'locate'], 'state_topic': 'valetudo/SilentTepidStinkbug/hass/SilentTepidStinkbug_vacuum/state', 'command_topic': 'valetudo/SilentTepidStinkbug/hass/SilentTepidStinkbug_vacuum/command', 'payload_start': 'START', 'payload_pause': 'PAUSE', 'payload_return_to_base': 'HOME', 'payload_stop': 'STOP', 'payload_locate': 'LOCATE', 'fan_speed_list': ['min', 'low', 'medium', 'high', 'max'], 'set_fan_speed_topic': 'valetudo/SilentTepidStinkbug/FanSpeedControlCapability/preset/set', 'unique_id': 'SilentTepidStinkbug_vacuum', 'availability_topic': 'valetudo/SilentTepidStinkbug/$state', 'payload_available': 'ready', 'payload_not_available': 'lost', 'device': {'manufacturer': 'Roborock', 'model': 'V1', 'identifiers': ['SilentTepidStinkbug'], 'sw_version': 'Valetudo 2023.12.0', 'configuration_url': 'http://valetudo-silenttepidstinkbug.local', 'name': 'Valetudo V1 SilentTepidStinkbug'}, 'name': 'Robot', 'platform': 'mqtt'} at discovery topic homeassistant/vacuum/SilentTepidStinkbug/SilentTepidStinkbug_vacuum/config
'''

[sca075]

@marvinwankersteen please see issue on MQTT Vacuum Camera 216, I just discovered that the problem was generated from the Camera in the end.
@jbouwh the issue was that I decoded the msg.payload on the connector.py and this created an overlap in memory address of the payload of MQTT. As per MQTT check if the payload is a type binary and then cover it, that in any cases now always fetch the real instance of the value, somehow there was some issue with it. I did correct the code avoiding to decode directly the payload and till now on the test environment looks okay. Will fix this on the next release of the camera scheduled for the end of this week.

[sca075]

@jbouwh
It looks I found the problem. If we use the default payload encoding (DEFAULT_ENCODING) the payload will be converted to string without any problem. Wile, binary data (zip files) are not correctly decoded (off course) and the warning raise -> it is a a subscribed payload so I don't know why this would influence the sensors in HA (as per locally it was handled correctly). While, subscribing the binary payload with None as encoding, all works.
In summary, to coply to the current implementation of MQTT in HA, DEFAULT_ENCODING must be used all the time, else b'value is warned / detected from MQTT. Meaning the Vacuums send the correct values (sensors are properly configured) but the only option to do not raise waning is to use the DEFAULT_ENCODING and no binary values in the Subscriptions.
This should definitely explain what happened.. and sorry but I don't know if this can help also to improve MQTT, probably documentation wise, if possible.

[jbouwh]

The encoding was explicitly set to ``. If the encoding option is left a way it will be utf-8, the default setting. That is also documented AFAIKT