SDS011 sensor data is sent on counterport, but this is not documented

[cyberman54]

Either document this, or better handle SDS011 data like user sensor data.

[proffalken]

Hey, is this still the case?

I've switched from Cayenne to Plain for various reasons and appear to have lost the SDS011 data even though the BLE/WiFi data continues to flow.

Looking at the decoder, I don't see anywhere that the SDS011 data is filtered out into the payload for port 1, so I'm assuming this needs to be added to the decoder somehow?

[proffalken]

@cyberman54 - apologies for the mention but just trying to work out the best way forward here.

Should I revert to Cayenne format and map the Cayenne name to the actual sensor name, or is there an easy fix for the plain decoder that will provide this data under the right name?

[cyberman54]

@proffalken could you put together a solution, and suggest it as PR here?

[proffalken]

@cyberman54 - I'd love too, but I'm struggling to know where to start.

I can see from the code that we store the values and then add it to the payload but I can't work out from that code where in the payload it ends up (i.e. which byte it is contained in)

If you (or someone else!) can point me in the right direction on working that bit out, I'm sure I can update the decoder for plain-text payloads accordingly, but my C++ isn't good enough to trace that all the way through and understand at which point of compiling the payyload this data is added.

[commeco]

I can see from the code that we store the values and then add it to the payload but I can't work out from that code where in the payload it ends up (i.e. which byte it is contained in)

Hi @proffalken,
the position in the payload depends on the configuration you have. E.g. if you use BLE, than the position is different. If you also use GPS, the position is +13 bytes etc. For the current master branch code, the byte Position in Port 1 are:

1-2 WiFi
+2 BLE (if BLE is configured)
+13 GPS (if GPS is configured)
+11 SDS011

So in your case without GPS the SDS011 data start at byte 5 in the payload.

@cyberman54
I'm thinking about working on a PR to implement two status bytes at the beginning of the payload which describe the data in the payload. Each bit of the status is equivalent to a specific sensor and a fix byte offset. For example:

0x0000 0000 0000 0001 WiFi    => 2-Byte
0x0000 0000 0000 0010 BLE     => 2-Byte
0x0000 0000 0000 0100 BME     => 8-Byte
0x0000 0000 0000 1000 GPS     => 13-Byte
0x0000 0000 0001 0000 SDS011  => 8-Byte

The order of the sensor values within the payload are fixed. If we have a config like 0x0000 0000 0001 0111 which means: WiFi, BLE, BME and SDS011

Than the payload byte positions are:

1-2   WiFi
3-4   BLE
5-12  BME
13-20 SDS011

In this scenario the decoder can handle the payload automatically. What do you think?

[cyberman54]

@commeco the problem with every change in the payload data structure is, that it would be a breaking change. So i think it's better not to touch it and solve this issue by adapting the decoder to the status quo of payload.

[proffalken]

@commeco - thanks, so the decoder needs to work out the length of the payload and then effectively "guess" where the values are based on how long it is and what it thinks is in there?

@cyberman54 - completely understand your concerns here, presumably this is why the Cayenne payload is an option, so that the fields always show up with the same labels? If that's the case then it may make more sense for me to create a "translation layer" in my code that says illuminance_21 is always PM2.5 (or whatever those labels actually are!)?

For reference, I'm working on https://github.com/proffalken/ttn2otel which takes the payload from TTN, discards the metadata, and converts the values into OpenTelemetry metrics that can then be forwarded on to a platform such as Grafana Cloud or DataDog for further analysis.

If I have control over the payload output, then I can just do a 1-1 mapping of the key pairs in the payload to the metrics, but if the names coming through are different to the data that is actually contained in the payload (which is the case with Cayenne as they don't have fields for PM2.5/PM10) then I need to add some kind of lookup, which I was hoping to avoid!

Appreciate the input from both of you on this!

[commeco]

@commeco the problem with every change in the payload data structure is, that it would be a breaking change. So i think it's better not to touch it and solve this issue by adapting the decoder to the status quo of payload.

Yes, you're right. Maybe i can solve this by a DEFINE key which distinguish between legacy payload structure and the new payload structure. Is that a way to go?

[commeco]

@commeco - thanks, so the decoder needs to work out the length of the payload and then effectively "guess" where the values are based on how long it is and what it thinks is in there?

@proffalken
Exactly, that's for the moment the "solution". In your case the additional lines in the decoder file look more or less like this:

// Add after the "if (bytes.length > 4)" block

    if (bytes.length >= 15) {
      decoded.sds011 = String.fromCharCode.apply(null, bytes[i]);
      i+=11;
    }

(Untested. :) )

[proffalken]

@commeco - thanks, so the decoder needs to work out the length of the payload and then effectively "guess" where the values are based on how long it is and what it thinks is in there?

@proffalken Exactly, that's for the moment the "solution". In your case the additional lines in the decoder file look more or less like this:

// Add after the "if (bytes.length > 4)" block

    if (bytes.length >= 15) {
      decoded.sds011 = String.fromCharCode.apply(null, bytes[i]);
      i+=11;
    }

(Untested. :) )

Thanks, I'll make that change in the next couple of days and play around with it, then commit it back once I've got it working.

[proffalken]

Finally managed to get around to this and with a bit of tweaking the code now looks as follows:

        if (input.bytes.length >= 15) {
          data.sds011 = String.fromCharCode.apply(null, input.bytes[i]);
          i+=11;
        }

Unfortunately, this is returning an error: TypeError: Value is not an object: undefined at apply (native)

Should the call to apply() set the type to bytes rather than null?

[proffalken]

@commeco - After some support from the TTN forum, I've got the following code that should decode this properly:

<!DOCTYPE html>
<html>
<head>
<title>Payload formatter tester</title>
</head>
<body>
<h1>Payload formatter tester</h2>

<p>SDS011: pm25: 1.20, pm10: 2.30</p>
<p>SDS011: pm25: 1.20, pm10: 4.20</p>

<h2>Input</h2>
<pre id="positions"></pre>
<pre id="payloadAsHex"></pre>
<p>Port: <span id="port"></span>
<p>Size: <span id="size"></span>

<h2>Result</h2>
<pre id="result"></pre>


<script>
// #### Your stuff here ####

const payloadFromConsole = "000200082C2020342E322C2020312E32";

const fPort = 1;


function decodeUplink(input) {
    var data = {};

    if (input.fPort === 1) {
        console.log("Port 1");
        var i = 0;

        if (input.bytes.length >= 2) {
            data.wifi = (input.bytes[i++] << 8) | input.bytes[i++];
            console.log("WiFi: "+ i);
        }
     
        if (input.bytes.length === 4 || input.bytes.length > 15) {
            data.ble = (input.bytes[i++] << 8) | input.bytes[i++];
            console.log("BLE: "+ i);
        }

        if (input.bytes.length >= 15) {
          data.sds011 = String.fromCharCode(input.bytes[i]);
          i++;
        }
        data.pax = 0;
        if ('wifi' in data) {
            data.pax += data.wifi;
        }
        if ('ble' in data) {
            data.pax += data.ble;
        } 
    }

    if (input.fPort === 2) {
        var i = 0;
        data.voltage = ((input.bytes[i++] << 8) | input.bytes[i++]);
        data.uptime = ((input.bytes[i++] << 56) | (input.bytes[i++] << 48) | (input.bytes[i++] << 40) | (input.bytes[i++] << 32) | (input.bytes[i++] << 24) | (input.bytes[i++] << 16) | (input.bytes[i++] << 8) | input.bytes[i++]);
        data.cputemp = input.bytes[i++];
        data.memory = ((input.bytes[i++] << 24) | (input.bytes[i++] << 16) | (input.bytes[i++] << 8) | input.bytes[i++]);
        data.reset0 = input.bytes[i++];
        data.restarts = ((input.bytes[i++] << 24) | (input.bytes[i++] << 16) | (input.bytes[i++] << 8) | input.bytes[i++]);
    }

    if (input.fPort === 4) {
        var i = 0;
        data.latitude = ((input.bytes[i++] << 24) | (input.bytes[i++] << 16) | (input.bytes[i++] << 8) | input.bytes[i++]);
        data.longitude = ((input.bytes[i++] << 24) | (input.bytes[i++] << 16) | (input.bytes[i++] << 8) | input.bytes[i++]);
        data.sats = input.bytes[i++];
        data.hdop = (input.bytes[i++] << 8) | (input.bytes[i++]);
        data.altitude = ((input.bytes[i++] << 8) | (input.bytes[i++]));
    }

    if (input.fPort === 5) {
        var i = 0;
        data.button = input.bytes[i++];
    }

    if (input.fPort === 7) {
        var i = 0;
        data.temperature = ((input.bytes[i++] << 8) | input.bytes[i++]);
        data.pressure = ((input.bytes[i++] << 8) | input.bytes[i++]);
        data.humidity = ((input.bytes[i++] << 8) | input.bytes[i++]);
        data.air = ((input.bytes[i++] << 8) | input.bytes[i++]);
    }

    if (input.fPort === 8) {
        var i = 0;
        if (input.bytes.length >= 2) {
            data.voltage = (input.bytes[i++] << 8) | input.bytes[i++];
        }
    }

    if (input.fPort === 9) {
        // timesync request
        if (input.bytes.length === 1) {
            data.timesync_seqno = input.bytes[0];
        }
        // epoch time answer
        if (input.bytes.length === 5) {
            var i = 0;
            data.time = ((input.bytes[i++] << 24) | (input.bytes[i++] << 16) | (input.bytes[i++] << 8) | input.bytes[i++]);
            data.timestatus = input.bytes[i++];
        }
    }

    if (data.hdop) {
        data.hdop /= 100;
        data.latitude /= 1000000;
        data.longitude /= 1000000;
    }

    data.bytes = input.bytes; // comment out if you do not want to include the original payload
    data.port = input.fPort; // comment out if you do not want to include the port

    return {
        data: data,
        warnings: [],
        errors: []
    };
}



// JS Payload support stuff - do NOT put this in to the TTS console

const toHexCount   = byteArray => 'Position:  ' + byteArray.map((x, i) => ('0' + i).slice(-2) ).join('   ');
const toHexStringX = byteArray => 'Payload: 0x' + Array.from(byteArray, byte => ('0' + byte.toString(16)).slice(-2)).join(' 0x')
function hexToBytes(hex) { let bytes = []; for (let c = 0; c < hex.length; c += 2) bytes.push(parseInt(hex.substr(c, 2), 16)); return bytes; }
var payload = {}
payload.bytes = hexToBytes(payloadFromConsole);
payload.fPort = fPort;
result = decodeUplink(payload);
console.log("data", result.data);
document.getElementById("positions").innerHTML = toHexCount(payload.bytes);	//payload.bytes.map((x, i) => ('0' + i).slice(-2) ).join('  ');
document.getElementById("payloadAsHex").innerHTML = toHexStringX(payload.bytes);
document.getElementById("port").innerHTML = payload.fPort;
document.getElementById("size").innerHTML = payload.bytes.length;
document.getElementById("result").innerHTML = JSON.stringify(result, null, 4);

</script>

</body>
</html>

Unfortunately, when given the payload 000200082C2020342E322C2020312E32 (which is the value in the TTN console once the payload has been sent by the PaxCounter), the output is as follows:

Payload formatter tester

(The next two lines are taken from the console output of the ESP32-Paxcounter itself)
SDS011: pm25: 1.20, pm10: 2.30

SDS011: pm25: 1.20, pm10: 4.20

Input
Position:  00   01   02   03   04   05   06   07   08   09   10   11   12   13   14   15
Payload: 0x00 0x02 0x00 0x08 0x2c 0x20 0x20 0x34 0x2e 0x32 0x2c 0x20 0x20 0x31 0x2e 0x32
Port: 1

Size: 16

Result
{
    "data": {
        "wifi": 2,
        "ble": 8,
        "sds011": ",",
        "pax": 10,
        "bytes": [
            0,
            2,
            0,
            8,
            44,
            32,
            32,
            52,
            46,
            50,
            44,
            32,
            32,
            49,
            46,
            50
        ],
        "port": 1
    },
    "warnings": [],
    "errors": []
}

I think I'm extracting the correct fields now because I've deleted the stuff that deals with GPS (I'm not using that here), but it's returning "," as a value, whereas I'd expect it to be one of the values from the console output.

Any help you can provide would be welcome, I'll be posting the same over on the TTN forums as well.

[commeco]

@proffalken
Is it possible to move to PAYLOAD_ENCODER == 2 ?
than it's quite simple to decode all the values.

[proffalken]

@proffalken Is it possible to move to PAYLOAD_ENCODER == 2 ? than it's quite simple to decode all the values.

It might be, but for now the following seems to work:

        if (input.bytes.length >= 15) {
          var sds011_tmp = "";
            while (i < 16) {
          sds011_tmp += String.fromCharCode(input.bytes[i]);
          i++;
            }
          data.pm10 = parseFloat(sds011_tmp.split(',')[1]);
          data.pm25 = parseFloat(sds011_tmp.split(',')[2]);
        }

So I'll leave it running for a bit and see if it gives the expected results, then I'll look at switching to another payload if that doesn't work.

[cyberman54]

@proffalken Is it possible to move to PAYLOAD_ENCODER == 2 ? than it's quite simple to decode all the values.

It might be, but for now the following seems to work:

        if (input.bytes.length >= 15) {
          var sds011_tmp = "";
            while (i < 16) {
          sds011_tmp += String.fromCharCode(input.bytes[i]);
          i++;
            }
          data.pm10 = parseFloat(sds011_tmp.split(',')[1]);
          data.pm25 = parseFloat(sds011_tmp.split(',')[2]);
        }

So I'll leave it running for a bit and see if it gives the expected results, then I'll look at switching to another payload if that doesn't work.

Thanks for investigating this. If it is working, could you please raise a PR for the adapted payload decoder?

I will then change it on TTN platform side for the predefined paxcounter device.

[proffalken]

Thanks for investigating this. If it is working, could you please raise a PR for the adapted payload decoder?

I will then change it on TTN platform side for the predefined paxcounter device.

It is working, but only if I remove the GPS code from the decoder.

I'm not using GPS on any of my devices, so it's not an issue for me, but I don't want to break someone else's setup.

I'm back at work now, and this is a "spare time" project, but I'll try and switch over to the Packed payload and see if that makes it easier to extract the data. If it does, I'll add back any changes that are needed there too, but it will probably be a few weeks before I get a chance to look at that.