Segment to Bucket Index for Volume Annotations

[fm3]

Creates an index structure for volume annotations. For every segment id, the list of buckets that contain this segment is stored. This is stored separately in the fossildb and updated on volume annotation mutation.

This lays the basis for new routes calculating properties of the segment volumes, e.g. the volume in voxels. A new route already provides that one. More complex metrics (surface area, sphericity, connected components) may be added later

TODO

• create data structure (proto vec3int list)
• add column family, put and get in fossildb
• on bucket save, get id set
• on bucket save, load previous bucket (either from fallback layer or volume), get id set
• get segment index for affected segments, mutate it, write back
• add route to compute total segment volume
• duplicate should update this index too
• downsampling should update this index too
• revert to version should update this index too (?)
• how to handle legacy volume annotations with no segment index? hasSegmentIndex → set to true in duplicate (if no fallback layer)
• perf optimization: skip previous bucket check for “duplicate”
• perf optimization: process multiple buckets at once (use local buffer + flush)
• perf optimization: avoid UnsignedInteger conversion of every voxel
  • for uint32
  • for the other types (ShortStream, ByteStream are not available)
• remove logging, clean up code
• merging volume tracings should update this index too
• add some explaining comments (e.g. to buffer)
• test some more (including merge)
• CI

Follow-up

• get fallback layer segment index if available (as computed by future versions of voxelytics?) segment-to-bucket index for volume annotations *with fallback segmentation* #7132
• perf optimization: fossildb multiget/multiput (with transaction management?), compare Multi-Get and Multi-Put with list of keys fossildb#39

Steps to test:

• Create volume annotation without fallback layer
• brush some, with several ids
• go to route /tracings/volume/<tracingId>/segmentStatistics/volume?segmentId=1&mag=1&token=<token>
• should get segment volume in vx
• merging annotations should still work (and create merged segment index if all source tracings have segment index)
• saving volume updates should not be unbearably slow (slower than before, though. I had about 100ms for 32 buckets, which is about thrice the old value)

Issues:

• fixes Segments-to-chunk index #7033

---

• Updated changelog
• Updated migration guide if applicable
• Removed dev-only changes like prints and application.conf edits
• Considered common edge cases
• Needs datastore update after deployment