Device database for simpler configuration

[postlund]

The problem:
There are a lot of devices and device types out there. With them comes a lot of different configurations ("datapoints"), making it difficult to configure a new device. Especially as a new user.

The solution:
To simplify set up of new devices, I want to create a database of devices with templates that can be used when setting up a device. Since a device can be set up in different ways, e.g. energy consumption as separate sensors or not, we might have to support multiple templates per device type.

The how:
I want to make this as simple as possible, using what we have:

• Device type can be identified via a product key, something we already discover in the config flow and save in the config entry. Multiple product keys can naturally map to the same configuration.
• As the database will evolve over time, it's not reasonable to distribute it with the component. The simplest solution would be to store it as a single file in the GitHub repo and download it when needed. For privacy reason this shall be opt-in. The downside is that the entire database will be downloaded instead of just configuration for the particular device. A better/future implementation would be to put a web service in front of the database and allow queries. I'm all up for doing this if someone wants to take the lead (it's not my field of expertise).
• Assuming we store the database as a single file, we can create a tool that allows us to take configuration on the command line and add it to the database. This will be used in the next step...
• We will need users to help us populate the database with their configuration. Again, for privacy reason, this is something that a user must actively do. I was thinking that we add a new service, e.g. localtuya.publish_config that creates a persistent notification containing links for all configured devices. Clicking one of the links will redirect the user to open an issue with the configuration for that device (or maybe single link for all devices). Some additional fields will have to be filled in by the user, e.g. device model and manufacturer. We can then automate adding it to the database using GitHub actions, by creating a workflow that parses the issue and opens a new PR with a commit where the device has been added (by calling the heller script created earlier).

There are a lot of considerations to take note of, but these are some of my initial thoughts of the overall design and how it would work. It's designed around being simple to implement in regards to infrastructure, as we don't need anything else than GitHub. The database will also be fully open and transparent, in case someone else would like to use it. Something I haven't talked about yet is database design and that would be next after finalizing the architecture as it is a minor thing.

[ultratoto14]

From what we already know, the configuration may be shared by many devices, so we may have two different ways of pushing a new device:

• Add the product key to an already known device configuration
• Add a full new device configuration

I have this kind of representation in mind for lights, what do you think:

{
    "ec6c5b6e-612d-4438-a31a-9cd452b6b614": {
        "productKeys": [
            "keyx7semxnwsecug",
            "keyahg5r8f5j4psn"
        ],
        "type": "light",
        "dps": {
            "id": 20,
            "color_mode": 21,
            "brightness": 22,
            "color_temp": 23,
            "color": 24,
            "scene": 25
        },
        "additional_data": {
            "brightness_upper": 1000,
            "color_temp_min_kelvin": 2200,
            "color_temp_max_kelvin": 4000,
            "scenes": {
                "Night": "000e0d0000000000000000c80000",
                "Read": "010e0d0000000000000003e801f4",
                "Meeting": "020e0d0000000000000003e803e8",
                "Leasure": "030e0d0000000000000001f401f4",
                "Soft": "04464602007803e803e800000000464602007803e8000a00000000",
                "Rainbow": "05464601000003e803e800000000464601007803e803e80000000046460100f003e803e800000000",
                "Shine": "06464601000003e803e800000000464601007803e803e80000000046460100f003e803e800000000",
                "Beautiful": "07464602000003e803e800000000464602007803e803e80000000046460200f003e803e800000000464602003d03e803e80000000046460200ae03e803e800000000464602011303e803e800000000"
            }
        }
    },
    "fc32a48e-34d3-4e91-9100-00f6c3c0142d": {
        "productKeys": [
            "keymhedstsqgpyrq"
        ],
        "type": "light",
        "dps": {
            "id": 1,
            "color_mode": 2,
            "brightness": 3,
            "color_temp": 4,
            "color": 5,
            "scene": 6
        },
        "additional_data": {
            "brightness_upper": 255,
            "color_temp_min_kelvin": 2700,
            "color_temp_max_kelvin": 6500,
            "scenes": {
                "Night": "bd76000168ffff",
                "Read": "fffcf70168ffff",
                "Meeting": "cf38000168ffff",
                "Leasure": "3855b40168ffff",
                "Scenario 1": "scene_1",
                "Scenario 2": "scene_2",
                "Scenario 3": "scene_3",
                "Scenario 4": "scene_4"
            }
        }
    }
}

[postlund]

We need to handle the fact that we can have several entities per device and I would also prefer to be able to add meatadata to each device, like manufacturer and model. Perhaps we can have three sections:

• Products: metadata (model, manufacturer m, etc.), pointer to table with possible datapoints and list of entities
• Datapoints: sets of datapoints and their description (potentially data types, ranges, etc)
• Entities: more or less the config we have today for each entity, but we only store a list of names for which datapoints should be looked up according to previous section (we can then re-use the same entity but use different datapoints for it)

An example using yaml (as that takes a little less space):

products:
  keyjcr45yfptp7h7:
    manufacturer: Deltaco
    model: SH-P01
    dps: id1
    entities: ["ent1"]

datapoints:
  id1:
    id:
      value: 1
      description: Power state
    current:
      value: 18
      description: Current (mA)

entities:
  ent1:
    platform: switch
    dps: ["id", "current"]

We can decide what id generation scheme to use later. Potentially we could adapt this scheme to handle multiple templates as well:

products:
  keyjcr45yfptp7h7:
     manufacturer: Deltaco
     model: SH-P01
     templates:
       - name: Super duper template
         dps: id1
         entities: ["ent1"]

Thoughts?

[ultratoto14]

Yes template is the key I think to help the user find the right options if his device is not supported.
So in this, where do you add entity attributes that are not DPs ?

What kind of db would you want to use ?

[postlund]

Yeah, with good descriptive names it can be good.

I would place other constants under the entity like this:

entities:
  ent1:
    platform: light
    dps: ["id"]
    brightness_lower: 25
    brightness_upper: 1000

As a simple start we could just store it as YAML (like above) and perhaps move it to a real database later (and add an API for it). We need other hosting capabilities for that though.

[ultratoto14]

Many devices will share the same configuration. I added the scenes data in my example because even now, this is the most significant part that could be different.

I'm ok that the yaml dB can be even in the repository at the beginning.

One last part, about the light of the diffuser we saw in one of the issues, do not see how to represent this kind of configuration that is very specific. Perhaps, different classes...

[postlund]

Yes, scenes are a good example of one configuration setting that can be different whilst the rest being same for many devices. It's possible to deal with this case too, of course, providing a look up table for scenes. The tricky part is that it becomes a special case and will likely be awkward to implement parsing for in a decent way (I mean parse user config and import to the database, we don't want to deal with special cases if we can avoid).

I'm not sure the diffuser is really different from any configuration where we create more than one entity (excluding trivial cases like dual socket plugs)? If there's no component in Home Assistant that matches the device, the best thing we can do is to split up the functionality into several entities and let the user decide how to deal with that, e.g. design UI in Lovelace. Not a big deal IMHO.

[ultratoto14]

For the diffuser, it's the way is the logic used for the light component. Changing the color_mode in ['white', 'colour', 'scene'] is not enough, a second DP should be changed with a value in ['1', '2', '3' ]. So the user in the flow, should have the ability to choose that 'logic' but most users should stay with the simple configuration. Do not know if we will have again different logics in the future.

About the DB, should we list all the available DPs in the device, even the one not useful. If we don't find the product key, maybe we can find a similar device by comparing the dp list, even without the values ?

For the scenes, we may check if the current value in the bulb is in some scene data set and propose these ones

[postlund]

But that's more at matter of how to configure and not to the database itself. The database is actually the solution to that problem since once we have a configuration in the database, it will be trivial to set up. It's the first time such a device is configured that is a bit cumbersome. If the user can't figure it out, hopefully it results in an issue here so we can help out and put it into the database afterwards.

I was thinking about storing all datapoints as a reference. Maybe we can make use of other datapoints in the future. One thing came to mine though, because of #183. In that issue, the reporter can manually set up the device via YAML. It's the DP detection that is broken. So here we can actually provide a solution via the database, but we won't have a complete dump of all datapoints. Should we mark that somehow?

One thing I was thinking... maybe we can create a pool of shared objects from the entities that we can re-use. Probably limit to lists and dicts. I was thinking something like:

entities:
  ent1:
    platform: light
    dps: ["id"]
    brightness_upper: 1000
    _scene: data1
  ent2:
    platform: light
    dps: ["id"]
    brightness_upper: 255
    _scene: data1

pool:
  data1:
    Night: 00ff00ff
    Day: ff00ff

Fake data obviously. I added _ in front of scene to indicate that it's value is in the pool (to not get in trouble with strings).

[postlund]

Maybe we should add a "type" field to the product, so we can classify the devices on a higher level (e.g. light, power plug, fan, etc). One idea I had was to allow manually picking a device from the database, in case current product key isn't there. Then we can suggest a similar device to try out (if we get an issue about it). By having both manufacturer and type we can provide better filtering to find devices.

[ultratoto14]

I like the idea of type that may mix different templates to build a product

[postlund]

That's not really how I was imagining it. A product is by itself a complete device. If you pick a template, that template should set up everything supported by that device. The templates themselves are just variations of how to do that.

My use case is more in the lines of buying an RGB light from vendor X which isn't in our database. The user could then filter out all other RGB lights from the same vendor, maybe finding a similar one that works. Perhaps it was just a new revision of an older light?

[ultratoto14]

With the tuya SDK i don't think you can do so different things, 95% of the manufacturer that will build a socket with power monitoring will use the same dp configuration.
For the lights (except the tuya diffuser 😄 ) it's the same, I have a led strip from one manufacturer that shares the same IDs and scene_data as a GU10 from another one.
I have GU10 from the same manufacturer of the led strip that use a completely different set of DP ids and scene_data.
I would say that, except a perfect match of product key, the dp id list is more efficient to identify a possible device compatibility.

But as soon as we have a "DB", we can use different way to request data in it, by manufacturer+type/by type+id list match and others.

[postlund]

Yeah, tuya has templates for all their products which also include default function sets (datapoints). A manufacturer can decide to use true default one or customize it, e.g. remove insured one or define custom ones. Usually DPS over 100 are custom by the manufacturer. But as you say, in the majority of cases we will share a lot of config between products by different manufacturers. Hopefully the design I've suggested will cover most cases in a good enough way. Otherwise we can change and migrate the data afterwards.

[junalmeida]

That database would be awesome. For supported devices, then discovery would be enough to add devices to HA.

[sibero80]

@postlund I Just went over the setup of a new diffuser, and it was a pain since there was no info on the Standard Instruction Set of the Device Debugging section, as it may be the case for a lot of cheap Chinese devices. but going through the process, I build a "heuristic" of some sort that might help this issue.

Here is an emulated user/localtuya flow.

1. User starts flow of adding new device in HASS.
2. User selects device from list.
3. Localtuya matches the devicename to the prebuilt database. If the Device exists on Known Device Database, then the template is used.
4. If the device does not exist on the Known Device Database, then the advertised name of the device is matched against the existing device taxonomy (light, dimmer, fan, difusser, plug, plug+powermeter, etc) Some of this work is already done by HASS, as each device class has a set of prebuilt entities that need to be populated.
5. If there is no match for the device, then the user is prompted to select the device type.
6. The user then is prompted with a set of entities associated to the device class and allowed to manually add more entities (Example, default template for a diffuser might not include a light entity, at this stage, user can add that entity manually).
7. User is prompted to provide the details for each entity as found on the Standard Instruction Set located at the Device Debugging section. Here user can add or modify entities. (Ideally user should be able to interact with each entity as it is added for faster troubleshooting).
8. If no device details are available on the Standard Instruction Set located at the Device Debugging section, then user would need to go through the TRAINING feature.
9. TRAINING feature. Here, the integration enters into a monitoring mode for all DP statuses (What I did to achieve this is to add the device and linked all DPs to the sensor entity to watch their state). Then, the user is prompted by LOCALTUYA to change as much settings of the device in the TUYA APP (this might be in a device-specific order, or randomly as if the user was training a new fingerprint on a mobile phone to maximize coverage). LOCALTUYA stores the different observed values of each DP during the user interactions.
10. After this "listening stage" LOCALTUYA has enough data for each entity to suggest pairing a DP with an entity type. User can pair a DP with an entity and try it out. The user is also allowed to manually modify the mapped values to add missing values not captured during the "listening stage".
11. User can save the configured device and share it to become part of the Known Device Database.

Apologies if this comment should go to a different issue, hopes this helps somehow. Wish I had some coding skills to help build something like this.