Cisco Model-Driven Telemetry (MDT) Input Plugin: Doesn't support/recognize Event-Driven Telemetry (EDT) Delete Messages

[severindellsperger]

Relevant telegraf.conf

[agent]
    interval = "10s"
    round_interval = true
    metric_buffer_limit = 10000
    flush_buffer_when_full = true
    collection_jitter = "0s"
    flush_interval = "10s"
    flush_jitter = "0s"
    debug = false
    quiet = false
    hostname = "telegraf"
[[inputs.cisco_telemetry_mdt]]
    transport = "grpc"
    service_address = ":32400"
[[outputs.kafka]]
    brokers = ["kafka:9092"]
    topic = "test.telemetry"
    data_format = "json"
[[outputs.file]]
    files = ["stdout"]
    data_format = "json"

Logs from Telegraf

# add ipv6 address on lo10
{"fields":{"state/ip":"10::10","state/origin":"STATIC","state/prefix_length":128,"state/status":"PREFERRED"},"name":"openconfig-interfaces:interfaces/interface/subinterfaces/subinterface/openconfig-if-ip:ipv6/addresses/address","tags":{"host":"telegraf","index":"0","ip":"10::10","name":"Loopback10","path":"openconfig-interfaces:interfaces/interface/subinterfaces/subinterface/openconfig-if-ip:ipv6/addresses/address","source":"sfo","subscription":"base_metrics"},"timestamp":1669633122}

# remove ipv6 on lo10 (this message is not shown on newer telemetry versions - it was removed in https://github.com/influxdata/telegraf/commit/dbd1fe734ba9222999cac2ad3cab5baf4023e6f2)
2022-11-28T10:58:27Z I! [inputs.cisco_telemetry_mdt] Message from sfo missing keys or content

System info

Telegraf 1.15-alpine (Docker), Ubuntu 22.04.1 LTS

Docker

The problem is not related to docker - anyway here is the docker-compose.yml:

version: "3"
services:
  zookeeper:
    image: 'bitnami/zookeeper:latest'
    ports:
      - '2181:2181'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: 'bitnami/kafka:latest'
    ports:
      - '9092:9092'
      - '9093:9093'
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
      - KAFKA_CFG_LISTENERS=CLIENT://:9092,EXTERNAL://:9093
      - KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka:9092,EXTERNAL://172.16.16.24:9093
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=CLIENT
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper
  telegraf:
    image: 'telegraf:1.15-alpine'
    ports:
      - '32400:32400'
    depends_on:
      - zookeeper
      - kafka

    volumes:
      - "./telegraf.conf:/etc/telegraf/telegraf.conf:ro"

Just to notice: the bug is also present in more recent telegraf version. But there the log message was removed, so we had to use the older version telegraf:1.15-alpine. For more details look at: dbd1fe7.

Steps to reproduce

1. Configure the cisco device with event-driven telemetry (on IOS-XR sample-interval 0):
   Here is an example with IOS-XR:

telemetry model-driven
 destination-group test
  address-family ipv4 xxx.xxx.xxx.xxx port 32400
   encoding self-describing-gpb
   protocol grpc no-tls
  !
 !
 sensor-group openconfig
  sensor-path openconfig-interfaces:interfaces/interface/subinterfaces/subinterface/openconfig-if-ip:ipv6/addresses/address
 !
 subscription base_metrics
 ! here the event-driven configuration (sample-interval 0)
 ! more info: https://www.cisco.com/c/en/us/td/docs/iosxr/ncs5500/telemetry/63x/b-telemetry-cg-ncs5500-63x/b-telemetry-cg-ncs5500-63x_chapter_010.html#id_36274
  sensor-group-id openconfig sample-interval 0  
  destination-id test
 !
!

2. On the router configure an example interface with ipv6:

RP/0/RP0/CPU0:sfo(config)#interface lo10 ipv6 address 10::10/128
RP/0/RP0/CPU0:sfo(config)#commit

3. Remove the ipv6 address of the interface with

RP/0/RP0/CPU0:sfo(config)#no interface lo10 ipv6 address 10::10/128
RP/0/RP0/CPU0:sfo(config)#commit

5. Observe the telemetry log message (only on older versions):

2022-11-28T11:10:52Z I! [inputs.cisco_telemetry_mdt] Message from sfo missing keys or content

...

Expected behavior

Telegraf should process the deleted message and create a metric with a name, tags and fields (possibly empty objects) and forward it to the configured output.

Actual behavior

The delete message from event-driven telemetry (EDT) is not processed at all. The logic is not implemented:

telegraf/plugins/inputs/cisco_telemetry_mdt/cisco_telemetry_mdt.go

Line 391 in c3562ae

if keys == nil || content == nil {

Additional info

No response

[srebhan]

@severindellsperger you might check the delete field of the message here and issue a "delete" metric. Not sure how it should look like (i.e. which fields it should contain) though...

[severindellsperger]

@severindellsperger you might check the delete field of the message here and issue a "delete" metric. Not sure how it should look like (i.e. which fields it should contain) though...

Exactly, this was the idea I had. I will take a look at it and keep you posted about it.

[severindellsperger]

Following is an example of how a metric could look when a delete message is sent.
It shows an example within the "openconfig-interfaces" model but could be similar in other models.
The important is that the "delete": "true" is set within the "fields" section.

{
  "fields": {
    "delete": "true"
  },
  "name": "openconfig-interfaces:interfaces/interface/subinterfaces/subinterface/openconfig-if-ip:ipv6/addresses/address",
  "tags": {
    "host": "telegraf",
    "index": "0",
    "ip": "10::10",
    "name": "Loopback10",
    "path": "openconfig-interfaces:interfaces/interface/subinterfaces/subinterface/openconfig-if-ip:ipv6/addresses/address",
    "source": "sfo",
    "subscription": "base_metrics"
  },
  "timestamp": 1669633122
}

[ubaumann]

Does the field key needs data, or could it be an empty object?

As a developer working with telemetry data, I prefer the data to be as static as possible. So I would like an empty object as fields and move the flag delete to tags.

So I would like to have something like:

{
  "fields": {},
  "name": "openconfig-interfaces:interfaces/interface/subinterfaces/subinterface/openconfig-if-ip:ipv6/addresses/address",
  "tags": {
    "host": "telegraf",
    "index": "0",
    "ip": "10::10",
    "name": "Loopback10",
    "path": "openconfig-interfaces:interfaces/interface/subinterfaces/subinterface/openconfig-if-ip:ipv6/addresses/address",
    "source": "sfo",
    "subscription": "base_metrics",
    "delete": "true"
  },
  "timestamp": 1669633122
}

The other option I would be okay with would be having the key delete in all the streaming data.

[severindellsperger]

We've discussed the following things in the slack channel.
It seems that it is not allowed to set an empty "fields object". Thus, we can not stick with the delete-tag solution.
So, from a developer's point of view, it seems to be the best solution if the delete info is always included in the fields section. As a result, there is always the same amount of elements, which is beneficial because some outputs would have problems otherwise.

Consequently, it would look as follows (see the delete tag in the fields section):

# add ipv6 address on lo10
{
  "fields": {
    "delete": false,
    "state/ip": "10::10",
    "state/origin": "STATIC",
    "state/prefix_length": 128,
    "state/status": "PREFERRED"
  },
  "name": "openconfig-interfaces:interfaces/interface/subinterfaces/subinterface/openconfig-if-ip:ipv6/addresses/address",
  "tags": {
    "host": "telegraf",
    "index": "0",
    "ip": "10::10",
    "name": "Loopback10",
    "path": "openconfig-interfaces:interfaces/interface/subinterfaces/subinterface/openconfig-if-ip:ipv6/addresses/address",
    "source": "sfo",
    "subscription": "base_metrics"
  },
  "timestamp": 1669633122
}

# remove ipv6 on lo10
{
  "fields": {
    "delete": true
  },
  "name": "openconfig-interfaces:interfaces/interface/subinterfaces/subinterface/openconfig-if-ip:ipv6/addresses/address",
  "tags": {
    "host": "telegraf",
    "index": "0",
    "ip": "10::10",
    "name": "Loopback10",
    "path": "openconfig-interfaces:interfaces/interface/subinterfaces/subinterface/openconfig-if-ip:ipv6/addresses/address",
    "source": "sfo",
    "subscription": "base_metrics"
  },
  "timestamp": 1669633122
}

I also think we should use the correct boolean values instead the strings "true" and "false" as defined in the JSON language.