Plugin

[domi4484]

Add plugin as it is now from https://github.com/teksi/wastewater_old
qgep qwat to ili is added as not submodule

[3nids]

what about git submodule?

qgis-plugin-ci easily package a submodule, that would simplify the packaging part.

[domi4484]

I can change to a submodule. I was a little scared by the fact that it is not possible to submodule only a part of a repository. It means that all parts of the library like tests, documentation, and so on will be checked out as well.
But in fact its not that much overhead.

[3nids]

git attributes allow to configure what we don't want in the archive, that's a good thing for packaging at least

[3nids]

do we need all the ili files in plugin/teksi_wastewater/tww2ili/ili ?

[sjib]

do we need all the ili files in plugin/teksi_wastewater/tww2ili/ili ?

1. All the model files are now available in the vsa and sia repositories

• https://www.vsa.ch/models
• https://405.sia.ch/models/

If we reference directly from there, the INTERLIS interface will automatically adjust to patch changes
If not, we have to manually adapt the model files each time and create a new release.
I see pros and cons in both approaches.

2. The readme.md does not need this text anymore - it should be deleted:
   License for use needed from VSA or sia: You can get it by purchasing Norm SIA405 2016 (http://www.webnorm.ch/null/null/sia%202016/d/2012/D/Product) or VSA-DSS CD (https://dss.vsa.ch/)

https://github.com/teksi/wastewater/blob/ab8b908141fbd8eeee3de0b3631797b0be7a10b8/plugin/teksi_wastewater/tww2ili/ili/README.md

[domi4484]

do we need all the ili files in plugin/teksi_wastewater/tww2ili/ili ?

In fact they can be retrieved automatically by the interlis tools. I'll remove them to avoid redundancy

[sjib]

@domi4484 I think you should still have a config somewhere which models are supported on the import/export.
On export it is defined within the export GUI, on import I have added the following check in the current version that supports also VSA-DSS and SIA405 Abwasser 2015:
https://github.com/QGEP/qgepqwat2ili/blob/3b5385403260ed67e994193e8eb93ec55f97ee87/qgepqwat2ili/utils/ili2db.py#L195

Maybe both (list on export and check on import) could be linked to the config - so code maintenace could be eased as when adding new models or changing releases, as only the configuration would need to be adapted and all other code relates to this. Else you have to know where else to adapt code in the plugin / gui / etc.

[domi4484]

@sjib for the import i think we can import both DSS and KEK to the in-between schema. At the moment of importing the XTF data we will get an error if the XTF contains additional unsupported models.

For the export do I unterstand it right that we need to choose what to export:

• KEK (+ SIA405)
• DSS (+ SIA405)
• only SIA405 as well?

[sjib]

Currently it is implemented on 2015 like this:

• KEK - creates two files - a VSA_KEK file including all network and TV data and a second file just with the network data SIA405 Abwasser (KEK inbetween schema is created and filled with all data and then two ili2db exports are run on the same intermediary schema with the two models VSA_KEK and in a second run SIA405 Abwasser)
• DSS - exporting all data export the KEK part in a DSS model
• SIA405 Abwasser just with the network data SIA405 Abwasser (this option is faster, as not all TV data will be exported in the intermediary schema)

See also my updated documentation: QGEP/docs#221

[sjib]

@domi4484
I have used the approach with 3 different mapping subfolders depending on the model: https://github.com/QGEP/qgepqwat2ili/tree/master/qgepqwat2ili/

[domi4484]

@sjib thanks for the explanations above!

When importing you xtf I see that there are many features with same combination of Bezeichnung and DatenherrRef
This collides with the index in the TEKSI model in_od_wastewater_structure_identifier who want this pair to be unique.

How should we handle that? Can the index be changed? Or should the data not contain such things?

One example in the xtf is the Bezeichnung 2.002-2.001 which is the same for objects ch080qwzHA000039 and ch080qwzKA000039

[sjib]

@domi4484

When importing you xtf I see that there are many features with same combination of Bezeichnung and DatenherrRef This collides with the index in the TEKSI model in_od_wastewater_structure_identifier who want this pair to be unique.

How should we handle that? Can the index be changed? Or should the data not contain such things?

One example in the xtf is the Bezeichnung 2.002-2.001 which is the same for objects ch080qwzHA000039 and ch080qwzKA000039

This constraint should only be the same as in the INTERLIS definition and not cover wastewater_structure and wastewater_network_element - but each separately:
UNIQUE !!@ comment = "UNIQUE Kombination Bezeichnung, DatenherrRef, damit mit VSA-DSS-Mini kompatibel (Wegleitung Daten der Siedlungsentwässerung 2020)" Bezeichnung, DatenherrRef; END Abwasserbauwerk;

and

UNIQUE 
    !!@ comment = "UNIQUE Kombination Bezeichnung, DatenherrRef, damit mit VSA-DSS-Mini kompatibel (Wegleitung Daten der Siedlungsentwässerung 2020)"
    Bezeichnung, DatenherrRef;
END Abwassernetzelement;

So no wastewater_structures with the same DatenherrRef / Bezeichnung.

But a similar wastewater_network_element (e.g. reach) is ok - and it is very common to name the reach the same as the channel.

Please list in_od_wastewater_structure_identifier here, so I can have a look at.

[domi4484]

It is defined like this:

CREATE UNIQUE INDEX in_od_wastewater_structure_identifier ON tww_od.wastewater_structure USING btree (identifier ASC NULLS LAST, fk_dataowner ASC NULLS LAST);

[domi4484]

In the case of Bezeichnung 1.001 there are elements of Deckel and Spezialbauwerk which are both wastewater structures in the end.

But in the case cited above the objects are Kanal -> ww Struktur and Haltung -> ww Network element so I don't understand the error in this case

[sjib]

No, Deckel is a subclass of structure_part and is associated to a wastewater_structure.

I suggest to try to try to turn off the index temporarely to check if it changes anything,
but from what I see https://www.postgresql.org/docs/current/indexes-unique.html it should do the same as the INTERLIS UNIQUE constraint.

[sjib]

@domi4484 Does the import in the intermediary schema works ok? On which step are you getting problems?

[domi4484]

I suggest to try to try to turn off the index temporarely to check if it changes anything,

If I turn it off everything works fine. The problem was happening on the last step, committing the transaction into tww_od.

No, Deckel is a subclass of structure_part and is associated to a wastewater_structure.

This is probably the point, in the tww_od representation for the mapping structure_part (and by consequence cover, benching, ...) was inheriting wastewater_structure.

Now I fixed that but I am having another issue on commit. Still investigating...

IntegrityError
(psycopg2.errors.ForeignKeyViolation) insert or update on table "structure_part" violates foreign key constraint "rel_structure_part_wastewater_structure"

DETAIL:  Key (fk_wastewater_structure)=(ch080qwzSW000125) is not present in table "wastewater_structure".

[SQL: INSERT INTO tww_od.structure_part (obj_id, identifier, remark, renovation_demand, last_modification, fk_dataowner, fk_provider, fk_wastewater_structure) VALUES (%(obj_id__0)s, %(identifier__0)s, %(remark__0)s, %(renovation_demand__0)s, %(last_modific ... 13160 characters truncated ... last_modification__72)s, %(fk_dataowner__72)s, %(fk_provider__72)s, %(fk_wastewater_structure__72)s)]
[parameters: {'last_modification__0': datetime.date(2019, 10, 14), 'obj_id__0': 'ch080qwzTF001272', 'fk_provider__0': 'ch080qwzPR000017', 'identifier__0': 'TF1272', 'fk_wastewater_structure__0': 'ch080qwzSW000125', 'renovation_demand__0': 137, 'fk_dataowner__0': 'ch080qwzPR000018', 'remark__0': None, 'last_modification__1': datetime.date(2019, 10, 14), 'obj_id__1': 'ch080qwzEF000009', 'fk_provider__1': 'ch080qwzPR000017', 'identifier__1': 'EF9', 'fk_wastewater_structure__1': 'ch080qwzSW000001', 'renovation_demand__1': 137, 'fk_dataowner__1': 'ch080qwzPR000018', 'remark__1': 'dringend reparieren', 'last_modification__2': datetime.date(2019, 10, 14), 'obj_id__2': 'ch080qwzEF000015', 'fk_provider__2': 'ch080qwzPR000017', 'identifier__2': 'EF15', 'fk_wastewater_structure__2': 'ch080qwzNS000013', 'renovation_demand__2': 138, 'fk_dataowner__2': 'ch080qwzPR000018', 'remark__2': None, 'last_modification__3': datetime.date(2019, 10, 14), 'obj_id__3': 'ch080qwzEF000024', 'fk_provider__3': 'ch080qwzPR000017', 'identifier__3': 'EF24', 'fk_wastewater_structure__3': 'ch080qwzNS000021', 'renovation_demand__3': 138, 'fk_dataowner__3': 'ch080qwzPR000018', 'remark__3': None, 'last_modification__4': datetime.date(2019, 10, 14), 'obj_id__4': 'ch080qwzEF000028', 'fk_provider__4': 'ch080qwzPR000017', 'identifier__4': 'EF28', 'fk_wastewater_structure__4': 'ch080qwzNS000026', 'renovation_demand__4': 5358, 'fk_dataowner__4': 'ch080qwzPR000018', 'remark__4': None, 'last_modification__5': datetime.date(2019, 10, 14), 'obj_id__5': 'ch080qwzEF000128', 'fk_provider__5': 'ch080qwzPR000017', 'identifier__5': 'EF128', 'fk_wastewater_structure__5': 'ch080qwzSW000125', 'renovation_demand__5': 138, 'fk_dataowner__5': 'ch080qwzPR000018', 'remark__5': None, 'last_modification__6': datetime.date(2019, 10, 14), 'obj_id__6': 'ch080qwzEF001004' ... 484 parameters truncated ... 'fk_dataowner__66': 'ch080qwzPR000018', 'remark__66': None, 'last_modification__67': datetime.date(2019, 10, 14), 'obj_id__67': 'ch080qwzDE001406', 'fk_provider__67': 'ch080qwzPR000017', 'identifier__67': 'EH2', 'fk_wastewater_structure__67': 'ch080qwzNS001406', 'renovation_demand__67': 5358, 'fk_dataowner__67': 'ch080qwzPR000018', 'remark__67': None, 'last_modification__68': datetime.date(2019, 10, 14), 'obj_id__68': 'ch080qwzDE001407', 'fk_provider__68': 'ch080qwzPR000017', 'identifier__68': 'VB214003', 'fk_wastewater_structure__68': 'ch080qwzNS001407', 'renovation_demand__68': 5358, 'fk_dataowner__68': 'ch080qwzPR000018', 'remark__68': None, 'last_modification__69': datetime.date(2019, 10, 14), 'obj_id__69': 'ch080qwzBN000010', 'fk_provider__69': 'ch080qwzPR000017', 'identifier__69': 'BN10', 'fk_wastewater_structure__69': 'ch080qwzSW000001', 'renovation_demand__69': 138, 'fk_dataowner__69': 'ch080qwzPR000018', 'remark__69': None, 'last_modification__70': datetime.date(2019, 10, 14), 'obj_id__70': 'ch080qwzBN000016', 'fk_provider__70': 'ch080qwzPR000017', 'identifier__70': 'BN16', 'fk_wastewater_structure__70': 'ch080qwzNS000013', 'renovation_demand__70': 137, 'fk_dataowner__70': 'ch080qwzPR000018', 'remark__70': None, 'last_modification__71': datetime.date(2019, 10, 14), 'obj_id__71': 'ch080qwzBN000025', 'fk_provider__71': 'ch080qwzPR000017', 'identifier__71': 'BN25', 'fk_wastewater_structure__71': 'ch080qwzNS000021', 'renovation_demand__71': 138, 'fk_dataowner__71': 'ch080qwzPR000018', 'remark__71': None, 'last_modification__72': datetime.date(2019, 10, 14), 'obj_id__72': 'ch080qwzBN000029', 'fk_provider__72': 'ch080qwzPR000017', 'identifier__72': 'BN29', 'fk_wastewater_structure__72': 'ch080qwzNS000026', 'renovation_demand__72': 5358, 'fk_dataowner__72': 'ch080qwzPR000018', 'remark__72': None}]
(Background on this error at: https://sqlalche.me/e/20/gkpj)

[sjib]

@domi4484 Could have a relation with the the Postgres / Postgis Version and missing z coordinates
We had some similar issues:

• Postgres 15 needs QGEP datamodel adaptations to work QGEP/QGEP#829
• Error upon import V - fk_wastewater_structure not present in table wastewater_structure QGEP/qgepqwat2ili#126
• Error upon import IV QGEP/qgepqwat2ili#125

Would be great if you can sort this out and would also give a hint for backporting.

[domi4484]

@sjib thanks it looks like the same error. I thought the statement SET CONSTRAINTS ALL DEFERRED; should avoid such errors but maybe it is something different...

I'll see if I can reduce the xtf to a minimal example to sort it out easier.

[sjib]

Maybe you can check how the imported geometries (*.Lage, Haltung.Verlauf, Abwasserbauwerk.Detailgeometrie) are handled and what happens, if they only have 2D coordinates x,y (as in the dataset). To what value is the z-value set in the datamodels 3D geometries that we have now? NULL, empty? Could this then conflict with any trigger/index/…?

[sjib]

@domi4484 I made a one pipe network - can you try this dataset?

If it still fails we can go back step by step - I have a whole series with

• just one manhole (no levels)
• one manhole - one special_structure (no levels)
• and so on with adding levels step by step

[sjib]

If you need I can also send the SIA405_Abwasser or VSA_KEK version instead.

[ponceta]

Tested on Windows on QGIS 3.28.13

• Wizard : Creation of wastewater structures
• Wizard : Creation of reaches
• Refresh network topology
• Connect wastewater elements
• Select upstream
• Select downstream
• Profile : starts well but python errors

Not tested (at least on my side) :

• INTERLIS import
• INTERLIS export

To be migrated to a draft plugin :

• datamodel tool -> https://github.com/orgs/teksi/discussions/87 (2024 Project : TEKSI Module Management Tool "TMMT")