Support 0d devices

[msillano]

I'm using tuyaApi in node-red, via node node-red-contrib-tuya-smart-device.
I have 20+ Tuya devices, like switches, sensors, ad some ZigBee devices. All devices work well in smartLife app.
The node-red actual comportment is:

1. All devices (except some sensors) accepts "SET" commands:

Example, using a real device, a switch, the command (sets the switch on for 10 s) :
payload: { multiple: true, data: { 1: true, 102: 10}}
produces the answer:
12 Dec 11:50:26 - [info] [tuya-smart-device:fe3fbeb2.ac31f] Data from device: "{"dps":{"1":true,"102":10},"t":1607770225}"

Using a virtual device, i.e. a radiator controller and ZigBee hub, the command (sets the target temperature)
payload: { "devId": "60a423fffeb5b90d", "dps": 103, "set": 220 }
produces the answer:
12 Dec 12:06:24 - [info] [tuya-smart-device:a9ea9736.7483f8] Data from device: "{"dps":{"103":220},"cid":"60a423fffeb5b90d","type":query,"t":1607771183}"

2. I get a message also when some values are changed in smartLife, e.g.:

12 Dec 12:14:08 - [info] [tuya-smart-device:fe3fbeb2.ac31f] Data from device: "{"dps":{"1":true},"t":1607771648}"
12 Dec 12:16:12 - [info] [tuya-smart-device:a9ea9736.7483f8] Data from device: "{"dps":{"101":47},"cid":"00158d00056e5022","t":1607771771}"

3. I get also messages in case of an event on the device, or from the sensors (without any request):

Switch countdown end:
12 Dec 11:50:36 - [info] [tuya-smart-device:fe3fbeb2.ac31f] Data from device: "{"dps":{"1":false,"102":0},"t":1607770235}"
Temperature from a ZigBee sensor, about every 3600s (request by Tuya cloud?):
12 Dec 12:14:28 - [info] [tuya-smart-device:a9ea9736.7483f8] Data from device: "{"dps":{"103":215},"cid":"60a423fffeb5b90d","t":1607771667}"

It sounds good, so all ok? Sorry, not really.

4. Any 'GET' command in any device (real, virtual, ...) in any form, don't work! Always I get:

12 Dec 11:32:32 - [error] [tuya-smart-device:fe3fbeb2.ac31f] json obj data unvalid

This is a big problem because I cannot poll any RO device data points, so some parameters are unreachable.

(see also issue #246 and issue #23)
I tryed also a tuyapi update to version 6, but no changes.

I found it very strange that the answer to a 'SET' command from node-red is always handled well, and the answer to a 'GET' command gets always an error (but the answer is well processed if the 'GET' is from the cloud).

Any idea?

Best regards
m.sillano

[Apollon77]

There are tuya devices that do not allow to query values via GET ... this is an device issue ... not this library. So they can only have values when they send them by themself. sorry

[codetheweb]

There are tuya devices that do not allow to query values via GET ... this is an device issue ... not this library. So they can only have values when they send them by themself. sorry

I don't think this is a thing - at least, I haven't ran into it.

What protocol version are you using? Have you tried both 3.1 and 3.3?

Might be related: #234

[Apollon77]

I also saw that from my users. The devices only push data, not possible to GET. Protocal was 3.3 in my examples when I remember correctly (protocol came from proxy sync still, so from official Tuya data)

[msillano]

Thanks for the comments...
I understand: the capacities of commercial devices are minimal.
I will test further on all my tuya devices.

But we have anyway a problem: the handling of the unallowed operations by device+tuyApi.

Example of unallowed operation:

• If I do a  SET to a not existing data point, the device does not send any response! (and TuyaApi does the same). 
  

• Same if I add to the input JSON parameters some EXTRA value. (e.g. payload= {"dps":1, "set":1, "extra" = 115})

For unallowed operations, a quiet abort is a correct strategy, not the message "json obj data unvalid" ( what json data???).

So a correct flow in tuyapi can be:

1. tests the JSON parameters for formal correctness, not for semantics, on input node data

• If data are a not valid JSON structure: ABORT, message "json obj data unvalid"
• if data valid: transmit data to device

2. on device answer

• no answer: quiet exit. no data output (maybe a INFO ?)
• correct data: returns JSON data
• response exists, but don't give a valid JSON structure: INFO with data hex dump + quiet exit, no data in output.

---

Abut "response exists, but don't produce a valid JSON structure" I fond that: device (sirene) response to a change (type) done by smartLive:

data: "3.34���o0�a�1� ���~�����q�����Q�S↵�����7nKC�K�������→����O���� ��6EU:̤sG�x��tpa" deviceId: "42027807d8bfc0c5831e" deviceName: "Sirena" excode: "332e33000000000000003400000001fdfd6f300861fd31fd20fdfdfd7e161610180671fdfdfdfd0451fd530afdfdfdfd1b376e4b43fd4b60fd6002fdfdfdfdfd09fdfdfdfd4ffdfd1efd20fdfd3645553a0c247347fd78fdfd747061"

The field "excode" is added by code:
const buf = Buffer.from(msg.payload.data, 'ascii'); msg.payload.excode =buf.toString("hex");

Very strange is the presence of many 0x00 bytes in a JSON value (not visible in the string !)
On same device, if I send a tentative {"operation":"SET", "dps":1, "set":1} via node-.red:

data: "�8e�cH[��*����QPjp|a��tt�D�k{���" deviceId: "42027807d8bfc0c5831e" deviceName: "Sirena" excode: "183865fd63485bfdfd2afdfdfdfd51506a707c61fd0e747410441e6b7bfdfd15"`

crazy behavior :)

Best regards
m.s.

[codetheweb]

I think there might be a misunderstanding. json obj data unvalid is sent from the device, not TuyAPI.

But there's a couple issues with doing it this way. First, silently failing is very rarely the best option; if an error occurs the user needs to know about it. More importantly, TuyAPI doesn't really have a way to check if a given payload is valid or not. There's no known schema to check against.

[msillano]

Thanks, codetheweb:, for your message.

1. I know that "json obj data unvalid" is sent from the device... But I said: device+tuyApi. A good API must overcome device limits, bugs, quirks. etc, to do a regular and optimal behavior to API users.

2. I know that API "does not know schema" (really, the general schema is known, the unknown is the schema used by a specific device). So the test on input must be only a validation test, not semantic against some specific schema ( i.e. the test checks only if the input object is a JSON object "well-formed").

3. consistency on API output and on error handling makes a better API, more easy to use.
   Remember: we have anyway a "silent failing" in case of extra data or bad DPS (on SET)! (i.e. in case of not correct JSON against the specific schema). But this is the same as using a 'GET' on a device not implementing it, so same handling!

Rules:

1. If the input is not a well Formed JSON structure --> ERROR "JSON obj not well-formed" (or some like this) (bad input parms : API aborts)
2. In the API output, the DPS field (if they exist) MUST always contain the set/get data point values (the new device status
3. In case of device error (no output at all, no JSON output, messages like "json obj data unvalid", not standard behavior, etc.)

• The API is without output, and sends an ERROR/INFO/DUMP trace messages,
• As an alternative, the API output presents the ERROR/INFO/DUMP message on a dummy JSON output, in special EXTRA fields, which may be useful for a specific recovery process.

So the programmer can write always:
if (output.payload.data.dps[1] !== undefined) status.on = output.payload.data.dps[1];

It is important to consider the difference between development (programming) errors, where the messages are helpful to the programmer, and runtime errors: in this case a "silent failing" is a valid option.

In hope to help create better tuyapi. :)

Best Regards
m.s.

[codetheweb]

( i.e. the test checks only if the input object is a JSON object "well-formed").

I'm having a hard time understanding what you mean by this, could you give an example?

It is important to consider the difference between development (programming) errors, where the messages are helpful to the programmer, and runtime errors: in this case a "silent failing" is a valid option.

Agreed, but hard to implement. Maybe could configure behavior based on NODE_ENV=[production|prod] if needed.

In general, this package is supposed to be a more low-level driver than a comprehensive solution. I'm happy for additional packages to be built on top of this that include built in device schemas, retry functionality, computed state, etc.

[msillano]

The tuyapi ver. "6.1.1" introduces two new options:

• options.nullPayloadOnJSONError = false
• options.issueGetOnConnect = true

Putting both to false is a partial solution to commands (like 'GET' and 'schema') not handled by some devices and occasionally reported by the device using the misunderstandly "json obj data invalid".

The only open issue is the 'ERROR' level in case of connection loss: using retry this event is not an ERROR requiring programmer intervention, better if it becomes a 'warning' (errors: "Error from socket", "find() timed out. Is the device powered on and the ID or IP correct?").

---

But I'm using the "node-red-contrib-tuya-smart-device" ver. 1.2.1, which uses tuyapi 5.3.1.
While waiting for an update, my personal solution is to modify "node_modules / tuyapi / index.js" (5.3.1) so:

1. comment out lines 317-332: the complete block: for (const packet of packets).
2. comment out line 404: this.get ();
3. changed line 623 to: throw new Error (` Retry connection: ID $ {this.device.id} `); more informative.

Best regards
m.s.

[codetheweb]

The only open issue is the 'ERROR' level in case of connection loss: using retry this event is not an ERROR requiring programmer intervention, better if it becomes a 'warning' (errors: "Error from socket", "find() timed out. Is the device powered on and the ID or IP correct?").

I'm a little confused on what you're saying here. Are you suggesting that TuyAPI should automatically attempt to re-connect after a 'bad' (user did not ask for it) disconnect?

[msillano]

That is the normal behavior, having many devices: some can disconnect for many reasons, like power down, interferences etc.
The nodes using tuyapi use an infinite retry strategy: the loss of connection is an event, handled by the application, not a fatal error.

Better if Tuyapi doesn't throw an error in case of 'bad' disconnect, but only info or warning.

                 29 Dec 09:17:31 - [info] [tuya-smart-device:ba1d97f9.86c698] Cannot find the device, re-trying...
                 29 Dec 09:17:32 - [info] [tuya-smart-device:ba1d97f9.86c698] initiating the find command
                 29 Dec 09:17:42 - [error] [tuya-smart-device:ba1d97f9.86c698] Error: Rety connection: ID 56685573d8bfc0508837
                 29 Dec 09:17:42 - [info] [tuya-smart-device:ba1d97f9.86c698] Cannot find the device, re-trying...
                 29 Dec 09:17:43 - [info] [tuya-smart-device:ba1d97f9.86c698] initiating the find command

---

Changing argument, how about protocol version 3.3 ? (see previous post).

best regards
m.s.

[presidentio]

It looks like I found the solution. You need to send just { cid: 'virtual_device_id' } got get all dps. virtual_device_id can be grabbed from any response

[Apollon77]

@presidentio interesting ... but what you mean with "can be grabbed from any response"? Is it also in the "discovery telegram"? or do we need to get at least one "pushed telegram"? is that cid identically to the devId or gwId?

[presidentio]

@Apollon77 To get cid you can connect to any device with enabled debug logging and update any state manually on the device or through the mobile application. In the logs you will see a json object. One of its fields is cid

[Apollon77]

Sure, but I search for an automated way of getting it ;-)

That's why my questions. Did you checked if the value is the same as devId or gwId?

[msillano]

Presentio > It looks like I found the solution. You need to send just { cid: 'virtual_device_id' }

That about virtual-devices, i.e. sub-devices, linked to tuya via a gateway.
I found, using a thermostat (SEA801-ZIGBEE Programmable Thermostat), a Zigbee gateway and tuya-smart-device using tuyapi6.1.1:

1. Changing the target temperature via smartLife app, I catch on tuya-smart-device the message:

       payload: object
         deviceId: "12373b1b789b5994cro7p"
         deviceName: "Zigbee Gateway"
         data: object
              cid: "890423fffeb5b90d"
              t: 1609852060
              dps: object
                      103: 200

This message is sent by the Zigbee Gateway, and cid identifies univocally the sub-device: the Thermostat#2.
Using _tuya-smart-device_, if I send this 'SET' payload to Gateway:

{
    "devId": "890423fffeb5b90d",
    "dps": "103",
    "set": 205
}

I get the same response message from the device!
(note: I MUST use the undocumented 'devId' parameter here).

This standard behavior is not the same for all sub-devices, all dps, all commands: SET/GET/MULTIPLESET/SCHEMA. In my experience, any device is someway special!

Testings with many devices I found 4 kinds of answers:

1. standard (JSON as expected)
2. none
3. the unclear message 'json obj data unvalid' (it looks like a default in the device internal parser, it can mind: "nothing to do, or because the required command is not implemented, or because the JSON is erroneous").
4. a binary undecoded message like " 3.34���o0�a�1� "

Best regards.
m.s.

[rospogrigio]

Hi everybody, I am sharing my experience with Tuya devices that will probably help you handle the "json data unvalid" problem. I am owner of an HA integration called localtuya, that handles communication Tuya devices locally, but without using tuyapi (see https://github.com/rospogrigio/localtuya ).
What we've discovered by combining source codes from different authors, is that there are 2 types of Tuya devices, that we call "type_0a" and "type_0d" devices, and they need to be queried in different ways in order to get the DPs values. 0x0a (DP_QUERY) and 0x0d (CONTROL_NEW) are respectively, the hex command that needs to be sent in order to retrieve the DPs.

1. type_0a devices require, as said and you know, the 0x0a command, with just this payload: {"gwId": "<deviceID>", "devId": "<deviceID>"} , and it will just return a json with the full list of DPs and their current value
2. type_0d devices are a bit trickier. If they are queried with the 0x0a command, they reply the well-known "json data unvalid" message. They require the 0x0d command instead, and a payload composed as follows:
   {"devId":"<deviceID>","uid":"<deviceID>","t":"<timestamp>","dps":{"1":null,"2":null,"100":null,"102":null,"103":null...}}
   Please note the "dps" list: this is a list of the DPs values you want to retrieve, pre-set at null; DP 1 should always be present. The device will reply with the current values of all the available DPs in the list. Since with a new device you might not know which DPs are available, when we want to autodetect them we just try requesting a list of all DPs from 1 to 30 and 100 to 120, since all devices encountered so far have DPs in this ranges. Unfortunely the payload string cannot exceed 255 chars, so we divide it into several requests of 10 DPs each.
   Finally, the last tricky bit is that these devices most of the times give an empty reply, BUT if you just repeat the recv() call, you finally get your reply with the DPs values.

@codetheweb I don't know how much of the above was of your knowledge, but I think it might help you review the code in order to support the "type_0d" devices too. Feel free to have a look at https://github.com/rospogrigio/localtuya/blob/master/custom_components/localtuya/pytuya/__init__.py and get inspiration.
Even with this, though, I am having difficulties communicating with zigbee wifi gateways so maybe we can help each other.
Let me have your thoughts, hope this can be helpful, bye!

[codetheweb]

Thanks for the info @rospogrigio, it ties together everything. Makes sense now.

Open to ideas on how to implement 0d devices. Should we query for DPS parameters automatically in connect(), or force the user to do so?

Also, do you know if there's a way to autodetect the device type @rospogrigio?

[Apollon77]

I think detection is exactly send get and if you get back "unvalid json" then remember the type.

And yes this should be done on get I think, so that it is irrelevant fpr the library user which type it is. you can remember on the first get what it is for all subsequent calls in with using this istance and/or also support an (optional) init flag so that you can diretcly pass in the type when you know from before.
Same: when you have the schema then you could also ad an init option where you can directly initialize the "Known" DPs and can so skip the auto detection

These infos are awesome

[Apollon77]

PS: hehe detection done as I proposed :-) https://github.com/rospogrigio/localtuya/blob/master/custom_components/localtuya/pytuya/__init__.py#L546

[rospogrigio]

I'm very glad to be helpful. Then you might be able to help me handle devices that communicate through a Wifi Zigbee gateway ;-) since I'm getting the same "unvalid json" and I don't know how to get past this...
Oh, and credits go to @tradeface , since he was the first that had the "0d" thing sorted out.

[codetheweb]

I don't currently have enough bandwidth to work on implementing 0d devices, but I'm happy to accept PRs. 😄

This issue is also getting kinda messy, so I started a new discussion here. Please move discussion about gateways there.

[rospogrigio]

@codetheweb you might be interested to read also what beatmag is reporting in rospogrigio/localtuya#194 , he has done some interesting discoveries too...

[codetheweb]

@BarryW55 I don't currently have a timeline on implementing support; partly because I don't have any 0d devices.

As always, happy to accept PRs. :)

[lknop]

@codetheweb I think I could get this to work, but as I am new to the party and js is not my weapon of choice, please confirm my assumptions about what should be done:

• the regular query should use DP_QUERY command as it does now
• if the response payload contains "data unvalid" (sic! 🤦) , we assume this is a 0d device and switch to the DPS command query
• possibly save this flag for the given device
• possibly give the user an option to set the flag manually upfront
• for a 0d device, use CONTROL_NEW command
• fix the response parser so that dps: { '1': true } payload is translated to true

Question: how to determine the actual dps payload to send? In one of the threads the following was proposed:
dps: { '1': null, '2': null, '3': null, '101': null, '102': null, '103': null }
EDIT: I have read up above about the workaround to query the dps ID in sets of 10. My question after reading the code is - should the subsequent calls be performed from within the get(options={}) method or should the api return after every request. This applies both to dps query and the switching to the CONTROL_NEW command.

[codetheweb]

possibly save this flag for the given device

Save for the current constructed instance yes, for future program executions no.

possibly give the user an option to set the flag manually upfront

Definitely.

should the subsequent calls be performed from within the get(options={}) method or should the api return after every request. This applies both to dps query and the switching to the CONTROL_NEW command.

If you're asking whether await get() should resolve before dps queries / switching to the 0d scheme is finished I'd say no.

Everything else sounds good, thanks for taking a look!

[BarryW55]

@iknop If there is anything I can do to help test, please let me know

[Apollon77]

No details, no follow up infos. closing. Please reopen if still relevant and happening with most recent versions.