Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable keeping duplicated geometries, but marking as "duplicated" with a property #9

Closed
robyngit opened this issue Sep 2, 2022 · 3 comments
Assignees
Labels
enhancement New feature or request pdg

Comments

@robyngit
Copy link
Member

robyngit commented Sep 2, 2022

For the IWP dataset, we decided to keep geometries that are classified as "duplicates" during processing in the version of the geopackage tiles that we archive. We will, however, mark duplicate geometries with a "duplicates" property. We therefore need to update the deduplication method to optionally keep duplicated geometries, but set a new "duplicated" property to True.

Duplicates should still be removed during the creation of the web products (3d tiles, PNG web tiles)

@robyngit
Copy link
Member Author

I am going to start working on this. I think we should restructure things a little so that:

  1. If there is a deduplication method set in the config, then duplicates are always identified during the staging process
  2. The deduplicate_at config option will be interpreted as the step(s) at which the duplicates should be removed (staging, raster, and/or 3dtiles)

robyngit added a commit that referenced this issue Oct 28, 2022
@robyngit
Copy link
Member Author

robyngit commented Oct 28, 2022

This feature is working in the feature-9-flag-duplicates branch. I've tested with both the neighbor and footprints method, but we will need to make updates to other parts of the workflow before we can implement this:

  • Update viz-raster to remove polygons flagged as duplicates, rather than run the deduplication method again
  • Update the StagedTo3DConverter to remove polygons flagged as duplicates, rather than run the deduplication method again
  • Update the merge script in the ray workflow to flag duplicates with the new methods when two staged tiles are merged

robyngit added a commit to PermafrostDiscoveryGateway/viz-raster that referenced this issue Nov 1, 2022
robyngit added a commit to PermafrostDiscoveryGateway/viz-workflow that referenced this issue Nov 1, 2022
robyngit added a commit to PermafrostDiscoveryGateway/viz-workflow that referenced this issue Nov 1, 2022
@robyngit
Copy link
Member Author

robyngit commented Nov 1, 2022

The new deduplication strategy is now merged into the main branch of all relevant packages. Briefly, with the new strategy:

  • Duplicate polygons are always identified during the staging step (and during the merging step, in the case that we are staging across multiple nodes and merging later)
  • During staging, a new boolean attribute (default name: staging_duplicated) is added that identifies polygons as duplicates or not
  • The deduplicate_at config option is interpreted to mean the step(s) at which polygons flagged as duplicates should be removed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request pdg
Projects
None yet
Development

No branches or pull requests

1 participant