BUG: timestamp management between Kafka and NGSI-LD #492
Comments
wagmarcel
added a commit
to wagmarcel/DigitalTwin
that referenced
this issue
Jan 9, 2024
Timestamp management was done so far by Kafka-timestamps. Every sample which entered the system throught MQTT-bridge is automatically stamped by Kafka. This ensured that the timestamp cannot be manipulated by future values and, thus, manipulating the system. The timestamp was stored in the TSDB but not explicitly in the NGSI-LD Database. This could lead to unclear timestamp interpretations. This PR makes timestamps more explicit by storing it also in the observedAt proerty in NGSI-LD. In addition, this PR provides a first draft of JuPyter-Lab notebooks for experimenting with Flink-SQL. In the past this was done by Apache Zeppelin but since there has not been any recent update the IFF Digital Twin team deciced to move to JuPyter. Related Issues: IndustryFusion#492 Signed-off-by: marcel <[email protected]>
abhijith-hr
pushed a commit
that referenced
this issue
Jan 9, 2024
Timestamp management was done so far by Kafka-timestamps. Every sample which entered the system throught MQTT-bridge is automatically stamped by Kafka. This ensured that the timestamp cannot be manipulated by future values and, thus, manipulating the system. The timestamp was stored in the TSDB but not explicitly in the NGSI-LD Database. This could lead to unclear timestamp interpretations. This PR makes timestamps more explicit by storing it also in the observedAt proerty in NGSI-LD. In addition, this PR provides a first draft of JuPyter-Lab notebooks for experimenting with Flink-SQL. In the past this was done by Apache Zeppelin but since there has not been any recent update the IFF Digital Twin team deciced to move to JuPyter. Related Issues: #492 Signed-off-by: marcel <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When samples enter through MQTT, they receive a Kafka timestamp. NGSI-LD has own timestamping methods like the system timestamps modifiedAt and createdAt. To align the timestamp concepts, the Kafka timestamp must be aligned with a NGSI-LD timestamp which can be set by API which is observedAt.
The text was updated successfully, but these errors were encountered: