Debugging missing data in ems-esp with help of km200 with use of API and Syslog

[tp1de]

Since I am using ems-esp and km200 in parallel, I ask myself what would be the most efficient / elegant way to debug missing fields in ems-esp while triggering changes on km200.

My idea would be: Within a javascript code writing changes to km200 and analyzing the respective telegrams by "filtering the right ones" out of syslog.

What do you think or what would be the easiest way to identify the right telegram types and content.

[proddy]

I would agree using syslog is the easiest way. With debug on and MQTT off (because there's a lot of messages). Then a script in python, JS or shell (using curl) to invoke the commands in batches every n seconds. What we could do is add a new API command that sets a marker in the syslog so you can see which commands are called when. Would that help?

It'll be good to get EMS-ESP closer to the KM200 and complete the missing library.

[tp1de]

What we could do is add a new API command that sets a marker in the syslog so you can see which commands are called when

Would it be possible to enable/disable syslog by API command?
I would then integrate the syslog server into javascript and then

1. enable syslog by API
2. change respective field by API via km200 Rest API (km200 gateway then sends ems message to device boiler/thermostat)
3. disable syslog by API
4. analyze syslog data -Which filters on syslog entries should be used? Do you have a proposal?

It'll be good to get EMS-ESP closer to the KM200 and complete the missing library.

I am ready to support.

[proddy]

yes that would work. I'll create a new API endpoint called 'system/syslog_level' which takes a json body where 'value' has the syslog level. So OFF is off and DEBUG is on in this case. Do you need some help with the scripting?

[tp1de]

Do you need some help with the scripting?

Thanks, not for the moment since I have all code already.
when switching from off to debug do you expect any time delay?

[tp1de]

@proddy just a question related to syslog messages set as hexadecimal bytes:

Is crc send as well as last byte? If yes: How is crc byte calculated?

First testings and resulting telegrams from gateway (48) to boiler (08). Value is send as hex in byte 5:

dhwCircuits.dhw1.temperatureLevels.high value:50 ---> 48 08 EA 06 32 19
dhwCircuits.dhw1.temperatureLevels.high value:52 ---> 48 08 EA 06 34 1F

dhwCircuits.dhw1.temperatureLevels.low value:45 ---> 48 08 EA 12 2D 2E
dhwCircuits.dhw1.temperatureLevels.low value:43 ---> 48 08 EA 12 2B 28

[proddy]

Is crc send as well as last byte? If yes: How is crc byte calculated?

If the option "Output EMS telegrams as hexadecimal bytes" is enabled then the last byte will be the CRC. I use a lookup table for speed in the EMS-ESP code (see telegram.cpp line 50).

[proddy]

First testings and resulting telegrams from gateway (48) to boiler (08). Value is send as hex in byte 5:
dhwCircuits.dhw1.temperatureLevels.high value:50 ---> 48 08 EA 06 32 19
dhwCircuits.dhw1.temperatureLevels.high value:52 ---> 48 08 EA 06 34 1F
dhwCircuits.dhw1.temperatureLevels.low value:45 ---> 48 08 EA 12 2D 2E
dhwCircuits.dhw1.temperatureLevels.low value:43 ---> 48 08 EA 12 2B 28

EMS-ESP uses EA (the telegram type ID) with offset 6 to write the wwSelTemp value on EMS+. So matching dhwCircuits.dhw1.temperatureLevels.high. Do we need a new command like wwSelTempLow for the low value at offset 12?

[tp1de]

EMS-ESP uses EA (the telegram type ID) with offset 6 to write the wwSelTemp value on EMS+. So matching dhwCircuits.dhw1.temperatureLevels.high

• YES for writing values - This is working like I mentioned
• NO for actual value set since it is reset to 60°C afterwards - Is one of the broadcasting messages overwriting value with wrong content?

[tp1de]

Do we need a new command like wwSelTempLow for the low value at offset 12?

I recommend to discuss this in a seperate thread later. I still struggle with your name conventions:
Within RC310 / km200 these are predefined temperature levels and selected temp (settemp) comes out of timeswitch programs and/or holiday modes. So names should be rather wwtemplevel_high and wwtemplevel_low then wwseltemp because selected temperature (wwsettemp) depends on time / holiday.

I will prepare an overview related to RC310 and EMS+ but it might take some time.

[tp1de]

I tried to test within a JS program but it turned out to be unnecessary complex.
I build a node-red workflow to test with filters for telegrams. This works fine and is quite easy.
So I am ready to do some tests related to km200 fields and the respective telegrams send.

I would like to test sending telegrams by myelf rather then just watching telegrams send by the km200 gateway.
So I have to find a JS way to generate the crc.

[proddy]

this code snippet will calculate the CR, if you can re-write in JS:

// Calculate CRC for buffer
uint8_t ems_crc( char * buffer, int len ){
 uint8_t i,crc = 0x0;
 uint8_t d;
 for(i=0;i<len-2;i++){
   d = 0;
   if ( crc & 0x80 ){
     crc ^= 12;
     d = 1;
   }
   crc  = (crc << 1)&0xfe;
   crc |= d;
   crc ^= buffer[i];
 }
 return crc;
}

[proddy]

I recommend to discuss this in a seperate thread later. I still struggle with your name conventions:
Within RC310 / km200 these are predefined temperature levels and selected temp (settemp) comes out of timeswitch programs and/or holiday modes. So names should be rather wwtemplevel_high and wwtemplevel_low then wwseltemp because selected temperature (wwsettemp) depends on time / holiday.

yes this needs looking at with Michael when he's back. Also why the value is been re-written each time.

Do you still need an API call to set the SysLog log level? Or add a marker?

[tp1de]

Do you still need an API call to set the SysLog log level? Or add a marker?

I think midterm it makes sense to be able to set SysLog level by API (marker does not make a lot of sense), but wait until we have discussed the other topics.

In the meatime I use node-red with the syslog-node in node-red and I am filtering the messages in node-red (e.g. from gateway to boiler) and then analyzing just the filtered messages. Whenever I change a km200 field value within my ioBroker adapter just one single telegram is send to boiler / thermostat. So it is easy to analyze within node-red for the moment.

Also why the value is been re-written each time.

Do you know in which broadcasting telegrams the wwseltemp is included?

[proddy]

wwseltemp is from telegram 0xE9 (offset 10 of the data block). There is also wwsettemp in 0xE9 at offset 0. It's confusing what the difference is.

[tp1de]

wwseltemp is from telegram 0xE9 (offset 10 of the data block).

Offset 10 of E9 is always 3C (60°C) and never change when I change the templevel_high. Just Offset 0 is changed but I guess that this would be rather actual settemp than temp_level_high. Since I do not find any value representing temp_level low in E9 I believe that E9 does not contain the data for temperature levels.

[tp1de]

BTW: Do you have an overview in respect to telegram types and respectiv values and their offset position for read / write as being used actually in ESP32 dev version?

[proddy]

BTW: Do you have an overview in respect to telegram types and respectiv values and their offset position for read / write as being used actually in ESP32 dev version?

Not really. It's all in code. I generated the read values for Boiler. The write ones need to be added manually which I'll do later.

boiler EMS mapping.txt

[tp1de]

Not really. It's all in code. I generated the read values for Boiler. The write ones need to be added manually which I'll do later.

Thanks, this is already a good start. 👍 I will do some checks ....

Addition: I think you filter unknown telegram types. So these are not in syslog just a message for unknown type. I do have approx 26 unknown telegram types and I think it might be usefull to have a look as well.
Most often unknown type is 8E4 -any idea what this could be?

Could you change SysLog so that they are in SysLog when hexadecimal bytes are selected?

[proddy]

the "No telegram type handler found for ID..." commands are printed in the log (Web, Console, Syslog) when the loglevel is DEBUG or higher. So setting this to DEBUG or ALL (or TRACE) for syslog should show you these alongside the hex telegram. I would need to check though but that's what I can remember from the code.

[tp1de]

the "No telegram type handler found for ID..." commands are printed in the log (Web, Console, Syslog) when the loglevel is DEBUG or higher. So setting this to DEBUG or ALL (or TRACE) for syslog should show you these alongside the hex telegram. I would need to check though but that's what I can remember from the code.

@proddy sorry but the telegram itself is not send just the message "No telegram type handler found for ID..."

It might be useful to introduce a loglevel "Telegrams" where just all telegrams on the EMS-Bus are send by syslog.
.... and if possible make change of SyslogLevel possible by API please.
(No text messages at all for MQTT, calling commands or missing type headers - just the hex telegrams on the EMS-Bus)

[proddy]

setting syslog level we can add - I made a new issue #98 to track it

changing the log behaviour to only show the telegrams is not possible due to how the cascaded Log Levels work (it's an industry standard). We need to think of another way. For now set the SysLog level to NOTICE and then in the console type watch raw. Only the telegrams will show in the SysLog

[tp1de]

I just see from Michaels syslog he posted in #90 the he gets the telegrams next to the message "No telegram type handler found for ID..." . Maybe I have made a mistake. Let me check again.

[tp1de]

Here is my SysLog for some short time, I can't find the respectiv telegrams to the message "No telegram type handler found for ID..."
Or are I am blind? Please have a look.

ems.log

[tp1de]

@proddy @MichaelDvP

Just a question for my understanding. If "No telegram type handler found for ID 0x8E4" is within log and I can't find telegrams with 8E4 within Syslog, can it be that these are telegrams with id 07 E4 ? (because I find them in syslog)

[MichaelDvP]

Yes, emsesp internally adds 0x100 to ems+ telegrams to differ between the type-id positions. The raw telegram 21 00 FF 00 01 55 is handled as type-id 0x255. That's because some devices use ems+telegrams with low numbers, like ISM using B0 00 FF 00 00 04.
If you use normal (not raw) logging the telegrams are shown with internal numbers.

[tp1de]

@MichaelDvP thanks for clarifying.

So this is valid for writing as well?
Does this mean that the internal type-id in ems-esp differs for ems+ telegrams from the type-id finally read/send on the ems-bus?

What about looking at other ems-bus wiki's on the net - Do I have to take this into account? (ems-esp type id's differ for ems+)

[MichaelDvP]

It affects only the logging in pretty format, the read-command and the watch-id.
All raw telegrams in logging, watching and sending uses the original ems-type-id, since the position is clearly defined.

[tp1de]

Thanks for clarifying. So console logging and syslog have same content.

[tp1de]

@proddy @MichaelDvP

The Syslog format is still wrong - even in last version 3.2.2b1.
Last working version was 3.2.1. where (hex and without) the Syslog messages had the right format.

With the versions afterwards and actual dev version only for rx messages format is OK.
All other message types have wrong format. Therefore reading Syslog with syslog node in node-red is getting a lot off error messages and the message buffer has an overflow after a certain time.

Any chances to get it corrected soon?

BTW: I have made a small ioBroker setup with a node-red flow analyzing and filtering the syslog and a javascript program to test all km200 writable fields with my ioBroker adapter to get in return the respective ems telegrams and write this info to a csv file.
I would like to test with the actual dev version and share the results with you afterwards.

[MichaelDvP]

The only change in syslog was #91, and this complies with specification. It's also independent of level and telegram format. I think there is something wrong with our node-red implementation.

[tp1de]

@MichaelDvP yes it might be related to the node-red-contrib-syslog-input node in node-red.
But since it was working before and now not anymore I am lost and stuck to fw 3.2.1.

What shall I do?

[tp1de]

@proddy @MichaelDvP
I think node-red syslog should work. Can this be re-established? It is ideal for fast prototyping.

In the meantime I installed v3.2.2b1 without node red and reading the syslog within javascript. But now I am confused because syslog content/format changed from previous versions.
Could you please explain the changes and content as per now?

[proddy]

I don't know or use node so can't help you there I'm afraid. It may be easier to capture some syslog on 3.2.1 and then compare it to 3.2.2b1. Without seeing real examples of what is not working its difficult to assist.

[tp1de]

Last changes of @MichaelDvP made syslog working again with node-red. Thanks a lot. 🥇

BTW: Are there already API commands for syslog available?

[proddy]

BTW: Are there already API commands for syslog available?

yes.

[tp1de]

Thanks but these are console commands.
What whould be the command to be send by rest-api to go to syslog-level "All" and switch on hex output?
...and what be the rest-api commands to switch syslog on / off?

[proddy]

@MichaelDvP looks like we need to start updating the wiki. I hate writing doc but i think its needed

[MichaelDvP]

In principal: Yes.
But in this case it does not help, first paragraph of api-section says that commands can be called from console, mqtt and api. Also a simple check of emsesp.local/api/system/commands would have helped.

[proddy]

I meant adding to the table here https://emsesp.github.io/docs/#/APIv3?id=system-endpoints with an explanation of the payload

[tp1de]

@proddy @MichaelDvP

I created a setup with syslog-server in a javascript with filter functions and then used my ioBroker adapter to change all writeable fields in km200 and ems-esp and discover the telegrams send on the ems-bus by analyzing the telegrams on syslog afterwards.

Let me share the result:
telegrams.xlsx
empty means no telegram send or htpp write error (even if marked as writeable)

Does this help?

[MichaelDvP]

Thanks,
i have expected more from km200, but a few values we can add to emsesp:
/heatingCircuits/hcx/fastHeatupFactor, as single byte uint
/dhwCircuits/dhw1/charge as bool
I think both writeable?

for the /hmx/hcMode, hmx/dhwMode and /heatingCircuits/hc2/switchProgramMode it is unclear why value and data differs.
and the switchPrograms has much values, but small data, that does not fit (or i do not understand it).
Only for hmx/startStop data and value match and can be read and set. Is it usefull without modes?

[tp1de]

I think both writeable?

Yes since I can test only fields marked writable - so all km200 fields are marked writeable by rest-api.
The switchprogram values are encoded in a special way I think. I will do some tests within the next days and report later.

[MichaelDvP]

The switchtime-coding is 2 bytes per switchtime, first byte upper 3 bits: day of week, lower 3 bits: mode, second byte time in steps of 15 min. SwitchProgramA, Data 03 14 01 58 FF FF FF FF FF FF FF FF is first: day 0, mode 3, 300 min, second: day 0, mode 1, 1320 min, all other times disabled. (btw. ems RC35 uses 10 min per step)

Why are the value/data for modes like /system/holidayModes/hm2/hcMode data 4, value 3 different? And what is the meaning of mode 3 or 4? What values are possible?

[tp1de]

The holiday modes hm1 .. hm5 are defined within rest-api:

1. assignedTo:

      "id": "/system/holidayModes/hm2/assignedTo",
      "type": "arrayData",
      "writeable": 1,
      "recordable": 0,
      "values": [],
      "allowedValues": [
        "hc1",
        "hc2",
        "hc3",
        "hc4",
        "dhw1",
        "dhw2"
      ],

2. dhwMode:

      "id": "/system/holidayModes/hm2/dhwMode",
      "type": "stringValue",
      "writeable": 1,
      "recordable": 0,
      "value": "OFF",
      "allowedValues": [
        "OFF",
        "TD_OFF"
      ],

3.hcMode:

      "id": "/system/holidayModes/hm2/hcMode",
      "type": "stringValue",
      "writeable": 1,
      "recordable": 0,
      "value": "ECO",
      "allowedValues": [
        "AUTO_SAT",
        "FIX_TEMP",
        "OFF",
        "ECO"
      ],

AUTO_SAT is defined as same as Saturday .... whatever this is for during holidays

4. startStop:

      "id": "/system/holidayModes/hm2/startStop",
      "type": "stringValue",
      "writeable": 1,
      "recordable": 0,
      "value": "2009-01-01/2009-01-01",
      "valIs": "value"

[tp1de]

@MichaelDvP @proddy
I recognized that for the datapoints /system/holidayModes/hm1...4/assignedTo the km200 sends 2 telegrams:
First one for hc1-hc4 and second one for dhw1 and dhw2.
I adjusted my analysis Excel table and added a colum for allowedValues / enum

telegrams.xlsx

[proddy]

closing.