Breaking change: Major rework of postgresql tables and kafka data str… #655
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…uctures The Flink SQL structure turned out to be too static to allow further developments. Therefore a redesign of Kafka Topics, database tables and Flink SQL rules were needed. After the redesign, more flexible attribute types, in particular: - Update of Flink SQL structures - removed attribute names from entity tables All attributes names are put into the same attributeValue field (instead of having dedicated object/value field depenend of attribute type. This allows adding in future GEO attributes for instance - Simplified Entities tables Now all entities are put into the same table (instead of having a table per entity) This allows easier SQL joins between entities and attributes - Database update to manage subattributes id and parentId are added to the tables to model sub and sub-sub attributes lang and unitType have been added to cover the respective NGSI-LD entities - Database update to manage arbitrary attribute types - Analogue to Flink updates the Postgresql database table attributes contains now a generic "attributeValue" field and lang and unitType fields. - In addition, added deleted field to entities, attribues and timescaledb tables to manage deleted attributes, entities and subelements more flexible - Redesign of core statementsets - To make the core-staementsets for forwarding ngsi-ld data to the respective alert topic - Update of Flink to version 1.19 - Kafka Topic updates - Attributes have now explicit synced field to detect whether there is an update necessary from attributes to Scorpio broker - All entities are now put to iff.ngsild.entities topic - lang and unitType fields are added according to the NGSI-LD specification Signed-off-by: marcel <[email protected]>
abhijith-hr
approved these changes
Dec 27, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…uctures
The Flink SQL structure turned out to be too static to allow further developments. Therefore a redesign of Kafka Topics, database tables and Flink SQL rules were needed. After the redesign, more flexible attribute types, in particular: