[Follow up]: Multiple segmentation layers

[philippotto]

Follow up for #5683.

Bugs & Weirdness:

• Toggling of segmentation layer via "3" does not work properly when there are multiple segmentation layers
  • how should we remember which layer was active before?
• "Create tracing" when viewing DS, simply picks visible segmentation layer as a base
  - [ ] also provide dropdown / modal as in dashboard to configure how the annotation looks
  • maybe default to new layer? <-- should be carefully considered
• Make hybrid simply picks visible segmentation layer as a fallback
• segmentationPatternOpacity is shared across all segmentation layers
• When having multiple segmentation layers and mesh files, the UI is a bit weird. For example, switching to another seg layer will hide loaded meshes in the meshes-tab, but they will still be visible in the 3D viewport.
• Enabling merger mode in a skeleton-only annotation when multiple segmentation layers exist and then switching between these layers will result in weird behavior, as the merger mode refers to the currently visible layer (done in Support multiple volume layers per annotation #5771; the merger mode remembers for which layer it was activated)

Refactoring:

• group currentMeshFileByLayer, availableMeshFilesByLayer, isosurfacesByLayer together in store
• mapping setup is complicated, since we only support one mapping at a time. getSegmentationLayerWithMappingSupport is used to listen on the layer for which the mapping should be bound on the GPU, but that function is messy, since it has to guess which layer is about to get used for a mapping. Currently, we cannot use isMappingEnabled since that is set to true at the very last stage of the mapping activation. We probably need another state (isBeingActivated) or something similar to avoid the guessing part. After refactoring test the following: open a dataset with two segmentation layers (both should have JSON mappings) and enable one mapping. Without the guessing logic, this simple step doesn't work right now.

More Features:

• Support having multiple volume annotation layers #5759
• Support showing multiple segmentation layers at once #5760
• Store visibility of layers per annotation (instead of per dataset) #5910
• Don't remove other volume layers when opening volume annotation #6143
• Allow to add a new volume tracing layer to an existing annotation #5855
• Add option to delete layers from an annotation #5802
  • then, remove the "unlinkFallbackSegmentation" feature
• Allow to import volume tracing into new annotation layer
• Always default to "non volume annotation" layer when creating an annotation. Instead, the user should be able to easily edit an existing segmentation layer ad-hoc (there could be an edit button for the layer). Note the following details:
  • the volume annotation tools should be shown even though no volume layer exists yet. clicking on them could ask the user which layer should be created.
  • adding a volume layer should not require a reload
• If multiple segmentation layers can be rendered, ensure that only one mapping can be enabled in the UI right now (as a mapping needs one additional texture we don't necessarily want to do this n>1 times).
  • this is a bit tricky because isEnabled in store is decoupled from the UI button. We need to ensure that getSegmentationLayerWithMappingSupport already returns the correct value quite early (maybe this is easier once Allow to share a tracing with activated mapping and loaded agglomerate skeletons via link #5738 is merged <-- it's still a bit tricky)

[normanrz]

"Create tracing" when viewing DS, simply picks visible segmentation layer as a base

• also provide dropdown / modal as in dashboard to configure how the annotation looks
  maybe default to new layer? <-- should be carefully considered

I think there should be one volume annotation layer per segmentation layer, so that each segmentation layer is editable.

[philippotto]

I think there should be one volume annotation layer per segmentation layer, so that each segmentation layer is editable.

I'd rather converge on a lazy-create approach, since the existence of a volume + fallback layer means that twice as many buckets are requested for that layer when viewing it. If the ad-hoc "add volume layer" functionality is smooth (i.e., does not require a reload), both approaches should/could feel very similar for the user.

[normanrz]

I think there should be one volume annotation layer per segmentation layer, so that each segmentation layer is editable.

I'd rather converge on a lazy-create approach, since the existence of a volume + fallback layer means that twice as many buckets are requested for that layer when viewing it. If the ad-hoc "add volume layer" functionality is smooth (i.e., does not require a reload), both approaches should/could feel very similar for the user.

Alternatively, we could create zero volume annotations and add an "edit" button to segmentation layers within an annotation.

[philippotto]

Yes, this sounds like a good idea. However, it's a bit more work (e.g., brush tools should be shown even though no volume layer exists yet; adding a layer should not require a reload etc.). I'll integrate this in the issue description, but I'd still solve the low hanging fruit first by adding a dropdown to the the "Create Annotation" button.

[kabilar]

Hi @philippotto @normanrz, is it currently possible to view multiple segmentation layers at once (#5760)? If not, what do you see as the major blockers? You have listed several items above, but just wanted to clarify since the issue was last updated a couple of years ago. Thank you.

cc @aaronkanzer @jingjingwu1225

[philippotto]

Hi @kabilar,

is it currently possible to view multiple segmentation layers at once (#5760)?

No, this is not possible currently.

If not, what do you see as the major blockers? You have listed several items above, but just wanted to clarify since the issue was last updated a couple of years ago.

The listed items above are not strictly related to #5760. So, none of these are blockers for it. However, the last checkpoint points out that only one segmentation mapping may be active currently. So, two segmentation layers with an enabled mapping cannot be rendered out of the box. This restriction would either needed to be lifted or properly respected when implementing the multi-segmentation-rendering issue.

[kabilar]

Thank you @philippotto. A couple of naive questions, as we would like to build out this feature on our fork. Happy to work with you to incorporate this upstream.

However, the last checkpoint points out that only one segmentation mapping may be active currently. So, two segmentation layers with an enabled mapping cannot be rendered out of the box.

What is a segmentation mapping?

This restriction would either needed to be lifted or properly respected when implementing the multi-segmentation-rendering issue.

What would it take to properly respect this mapping?

[aaronkanzer]

Hi @philippotto @fm3 @normanrz --- just wanted to follow up per @kabilar 's message there

I'm noticing a handful of application logic in oxalis that pertains to rendering of segmentation layers -- any chance you could provide some context for why n+1 segmentation layers can't be rendered at once?

Thanks again for the time and consideration!

[philippotto]

From the top of my head, here are a few pointers:

• all user interactions with segmentations currently refer to the currently visible segmentation layer (brushing, hovering/selecting segments etc). this would need to change if multiple segmentations are visible.
• blending for segmentation layers would need some tuning (the default "additive" blending does not make sense for segmentations; there's already "cover" which would need to be used always for segmentations)
• segmentation mappings are basically dictionaries which map the original segment id to another (e.g., when segments 1 and 2 should be merged, the mapping would be {1: 1, 2: 1}). the data of the mapping needs to be available on the gpu for the rendering to work (currently, only one is bound to the GPU but binding multiple ones should be possible in general if enough texture slots are available).