Rework volume interpolation feature to be shortcut-bound and support depths > 2

CHANGELOG.unreleased.md

@@ -19,6 +19,7 @@ For upgrade instructions, please check the [migration guide](MIGRATIONS.released
 - Added segmentation layers to the functionality catching the case that more layers are active that the hardware allows. This prevents rendering issue with more than one segmentation layer and multiple color layers. [#6211](https://github.com/scalableminds/webknossos/pull/6211)
 - Selecting "Download" from the tracing actions now opens a new modal, which lets the user select data for download, start TIFF export jobs or copy code snippets to get started quickly with the webKonossos Python Client. [#6171](https://github.com/scalableminds/webknossos/pull/6171)
 - Adding a New Volume Layer via the left border tab now gives the option to restrict resolutions for the new layer. [#6229](https://github.com/scalableminds/webknossos/pull/6229)
+- Added support for segment interpolation with depths > 2. Also, the feature was changed to work on an explicit trigger (either via the button in the toolbar or via the shortcut V). When triggering the interpolation, the current segment id is interpolated between the current slice and the least-recently annotated slice. [#6235](https://github.com/scalableminds/webknossos/pull/6235)
 
 ### Changed
 - When creating a new annotation with a volume layer (without fallback) for a dataset which has an existing segmentation layer, the original segmentation layer is still listed (and viewable) in the left sidebar. Earlier versions simply hid the original segmentation layer. [#6186](https://github.com/scalableminds/webknossos/pull/6186)
@@ -35,5 +36,6 @@ For upgrade instructions, please check the [migration guide](MIGRATIONS.released
 - Fixed a bug which could cause a segmentation layer's "ID mapping" dropdown to disappear. [#6215](https://github.com/scalableminds/webknossos/pull/6215)
 
 ### Removed
+- Removed the feature to copy a segment from the previous/next slice with the V shortcut. Use the new volume interpolation feature instead (also bound to V and available via the toolbar). [#6235](https://github.com/scalableminds/webknossos/pull/6235)
 
 ### Breaking Changes

docs/keyboard_shortcuts.md

@@ -102,8 +102,7 @@ Note that you can enable *Classic Controls* which will behave slightly different
 | C                                 | Create New Segment                                             |
 | W                                 | Toggle Modes (Move / Skeleton / Trace / Brush / ...)        |
 | SHIFT + Mousewheel or SHIFT + I, O | Change Brush Size (Brush Mode)                             |
-| V                                 | Copy Segmentation of Current Segment From Previous Slice       |
-| SHIFT + V                         | Copy Segmentation of Current Segment From Next Slice           |
+| V                                 | Interpolate current segment between last labeled and current slice |
 
 Note that you can enable *Classic Controls* which won't open a context menu on right-click, but instead erases when the brush/trace tool is activated.
 
@@ -132,4 +131,4 @@ The following binding only works in skeleton/hybrid annotations and if an agglom
 Note that you can enable *Classic Controls* in the left sidebar. 
 Classic controls are provided for backward compatibility for long-time users and are not recommended for new user accounts.
 Hence, Classic controls are disabled by default, and webKnossos uses a more intuitive behavior which assigns the most important functionality to the left mouse button (e.g., moving around, selecting/creating/moving nodes). The right mouse button always opens a context-sensitive menu for more complex actions, such as merging two trees.
-With classic controls, several mouse controls are modifier-driven and may also use the right-click for actions, such as erasing volume data.
+With classic controls, several mouse controls are modifier-driven and may also use the right-click for actions, such as erasing volume data.

docs/volume_annotation.md

@@ -73,15 +73,14 @@ Due to performance reasons, 3D flood-fills only work in a small, local bounding
 Check the `Processing Jobs` page from the `Admin` menu at the top of the screen to track progress or cancel the operation. The finished, processed dataset will appear as new dataset in your dashboard.
 
 ### Volume Interpolation
-When using the brush or trace tool, you can enable `Volume Interpolation` for faster annotation speed (in a task context, this feature has to be enabled explicitly).
-When enabled, it suffices to only label every second slice. The skipped slices will be filled automatically by interpolating between the labeled slices.
+When using the brush or trace tool, you can use the `Volume Interpolation` feature for faster annotation speed (in a task context, this feature has to be enabled explicitly).
+Simply label a segment in one slice (e.g., z=10), move forward by a few slices (e.g., z=14) and label the segment there.
+Now, you can click the "Interpolate" button (or use the shortcut V) to interpolate the segment between the annotated slices (e.g., z=11, z=12, z=13).
 
 Note that it is recommended to proof-read the interpolated slices afterwards, since the interpolation is a heuristic.
 
 ![Video: Volume Interpolation](https://www.youtube.com/watch?v=-nYv0hA1k4A)
 
-The little arrow at the interpolation button in the toolbar indicates whether you are currently labeling with increasing or decreasing X/Y/Z.
-
 ### Mappings / On-Demand Agglomeration
 With webKnossos it is possible to apply a precomputed agglomeration file to re-map/combine over-segmented volume annotations on-demand. Instead of having to materialize one or more agglomeration results as separate segmentation layers, ID mappings allow researchers to apply and compare different agglomeration strategies of their data for experimentation. 
 

frontend/javascripts/admin/tasktype/recommended_configuration_view.tsx

@@ -49,7 +49,6 @@ function getRecommendedConfigByCategory() {
     },
     volume: {
       brushSize: 50,
-      isVolumeInterpolationEnabled: false,
     },
   };
 }

frontend/javascripts/libs/mjs.ts

@@ -282,6 +282,15 @@ const V3 = {
   toArray(vec: ArrayLike<number>): Vector3 {
     return [vec[0], vec[1], vec[2]];
   },
+
+  roundElementToResolution(vec: Vector3, resolution: Vector3, index: 0 | 1 | 2): Vector3 {
+    // Rounds the element at the position referenced by index so that it's divisible by the
+    // resolution element.
+    // For example: roundElementToResolution([11, 12, 13], [4, 4, 2], 2) == [11, 12, 12]
+    const res: Vector3 = [vec[0], vec[1], vec[2]];
+    res[index] = Math.floor(res[index] / resolution[index]) * resolution[index];
+    return res;
+  },
 };
 
 export { M4x4, V2, V3 };

frontend/javascripts/libs/utils.ts

@@ -915,3 +915,13 @@ export function coalesce<T>(obj: { [key: string]: T }, field: T): T | null {
   }
   return null;
 }
+
+export function pluralize(str: string, count: number, optPluralForm: string | null = null): string {
+  if (count < 2) {
+    return str;
+  }
+  if (optPluralForm != null) {
+    return optPluralForm;
+  }
+  return `${str}s`;
+}

frontend/javascripts/messages.ts

@@ -43,7 +43,6 @@ export const settings: Partial<Record<keyof RecommendedConfiguration, string>> =
   gpuMemoryFactor: "Hardware Utilization",
   overwriteMode: "Volume Annotation Overwrite Mode",
   useLegacyBindings: "Classic Controls",
-  isVolumeInterpolationEnabled: "Volume Interpolation",
 };
 export const settingsTooltips: Partial<Record<keyof RecommendedConfiguration, string>> = {
   loadingStrategy: `You can choose between loading the best quality first

frontend/javascripts/oxalis/controller/viewmodes/plane_controller.tsx

@@ -30,7 +30,7 @@ import Toast from "libs/toast";
 import * as Utils from "libs/utils";
 import {
   createCellAction,
-  copySegmentationLayerAction,
+  interpolateSegmentationLayerAction,
 } from "oxalis/model/actions/volumetracing_actions";
 import { cycleToolAction } from "oxalis/model/actions/ui_actions";
 import {
@@ -130,10 +130,7 @@ class VolumeKeybindings {
     return {
       c: () => Store.dispatch(createCellAction()),
       v: () => {
-        Store.dispatch(copySegmentationLayerAction());
-      },
-      "shift + v": () => {
-        Store.dispatch(copySegmentationLayerAction(true));
+        Store.dispatch(interpolateSegmentationLayerAction());
       },
     };
   }

frontend/javascripts/oxalis/default_state.ts

@@ -76,8 +76,6 @@ const defaultState: OxalisState = {
     overwriteMode: OverwriteModeEnum.OVERWRITE_ALL,
     fillMode: FillModeEnum._2D,
     useLegacyBindings: false,
-    isVolumeInterpolationEnabled: false,
-    volumeInterpolationDepth: 2,
   },
   temporaryConfiguration: {
     viewMode: Constants.MODE_PLANE_TRACING,

frontend/javascripts/oxalis/model/accessors/volumetracing_accessor.ts

@@ -11,6 +11,7 @@ import type {
 import type {
   ActiveMappingInfo,
   HybridTracing,
+  LabelAction,
   OxalisState,
   SegmentMap,
   Tracing,
@@ -27,12 +28,14 @@ import {
   getVisibleSegmentationLayer,
 } from "oxalis/model/accessors/dataset_accessor";
 import { getMaxZoomStepDiff } from "oxalis/model/bucket_data_handling/loading_strategy_logic";
-import { getRequestLogZoomStep } from "oxalis/model/accessors/flycam_accessor";
+import { getFlooredPosition, getRequestLogZoomStep } from "oxalis/model/accessors/flycam_accessor";
 import { reuseInstanceOnEquality } from "oxalis/model/accessors/accessor_helpers";
+import { V3 } from "libs/mjs";
 
 export function getVolumeTracings(tracing: Tracing): Array<VolumeTracing> {
   return tracing.volumes;
 }
+
 export function getVolumeTracingById(tracing: Tracing, tracingId: string): VolumeTracing {
   const volumeTracing = tracing.volumes.find((t) => t.tracingId === tracingId);
 
@@ -42,10 +45,12 @@ export function getVolumeTracingById(tracing: Tracing, tracingId: string): Volum
 
   return volumeTracing;
 }
+
 export function getVolumeTracingLayers(dataset: APIDataset): Array<APISegmentationLayer> {
   const layers = getSegmentationLayers(dataset);
   return layers.filter((layer) => layer.tracingId != null);
 }
+
 export function getVolumeTracingByLayerName(
   tracing: Tracing,
   layerName: string,
@@ -55,14 +60,17 @@ export function getVolumeTracingByLayerName(
   const volumeTracing = tracing.volumes.find((t) => t.tracingId === layerName);
   return volumeTracing;
 }
+
 export function hasVolumeTracings(tracing: Tracing): boolean {
   return tracing.volumes.length > 0;
 }
+
 export function getVolumeDescriptors(
   annotation: APIAnnotation | APIAnnotationCompact | HybridTracing,
 ): Array<AnnotationLayerDescriptor> {
   return annotation.annotationLayers.filter((layer) => layer.typ === "Volume");
 }
+
 export function getVolumeDescriptorById(
   annotation: APIAnnotation | APIAnnotationCompact | HybridTracing,
   tracingId: string,
@@ -77,6 +85,7 @@ export function getVolumeDescriptorById(
 
   return descriptors[0];
 }
+
 export function getReadableNameByVolumeTracingId(
   tracing: APIAnnotation | APIAnnotationCompact | HybridTracing,
   tracingId: string,
@@ -115,10 +124,12 @@ export function getServerVolumeTracings(
   );
   return volumeTracings;
 }
+
 export function getActiveCellId(volumeTracing: VolumeTracing): number {
   const { activeCellId } = volumeTracing;
   return activeCellId;
 }
+
 export function getContourTracingMode(volumeTracing: VolumeTracing): ContourMode {
   const { contourTracingMode } = volumeTracing;
   return contourTracingMode;
@@ -137,6 +148,7 @@ const MAG_THRESHOLDS_FOR_ZOOM: Partial<Record<AnnotationTool, number>> = {
 export function isVolumeTool(tool: AnnotationTool): boolean {
   return VolumeTools.indexOf(tool) > -1;
 }
+
 export function isVolumeAnnotationDisallowedForZoom(tool: AnnotationTool, state: OxalisState) {
   if (state.tracing.volumes.length === 0) {
     return true;
@@ -171,12 +183,14 @@ export function getMaximumBrushSize(state: OxalisState) {
   // we double the maximum brush size.
   return MAX_BRUSH_SIZE_FOR_MAG1 * 2 ** lowestExistingResolutionIndex;
 }
+
 export function isSegmentationMissingForZoomstep(
   state: OxalisState,
   maxZoomStepForSegmentation: number,
 ): boolean {
   return getRequestLogZoomStep(state) > maxZoomStepForSegmentation;
 }
+
 export function getRequestedOrVisibleSegmentationLayer(
   state: OxalisState,
   layerName: string | null | undefined,
@@ -230,9 +244,11 @@ export function getRequestedOrDefaultSegmentationTracingLayer(
 
   return getTracingForSegmentationLayer(state, visibleLayer);
 }
+
 export function getActiveSegmentationTracing(state: OxalisState): VolumeTracing | null | undefined {
   return getRequestedOrDefaultSegmentationTracingLayer(state, null);
 }
+
 export function getActiveSegmentationTracingLayer(
   state: OxalisState,
 ): APISegmentationLayer | null | undefined {
@@ -244,6 +260,7 @@ export function getActiveSegmentationTracingLayer(
 
   return getSegmentationLayerForTracing(state, tracing);
 }
+
 export function enforceActiveVolumeTracing(state: OxalisState): VolumeTracing {
   const tracing = getActiveSegmentationTracing(state);
 
@@ -253,6 +270,7 @@ export function enforceActiveVolumeTracing(state: OxalisState): VolumeTracing {
 
   return tracing;
 }
+
 export function getRequestedOrVisibleSegmentationLayerEnforced(
   state: OxalisState,
   layerName: string | null | undefined,
@@ -268,13 +286,15 @@ export function getRequestedOrVisibleSegmentationLayerEnforced(
     "No segmentation layer is currently visible. Pass a valid layerName (you may want to use `getSegmentationLayerName`)",
   );
 }
+
 export function getNameOfRequestedOrVisibleSegmentationLayer(
   state: OxalisState,
   layerName: string | null | undefined,
 ): string | null | undefined {
   const layer = getRequestedOrVisibleSegmentationLayer(state, layerName);
   return layer != null ? layer.name : null;
 }
+
 export function getSegmentsForLayer(
   state: OxalisState,
   layerName: string | null | undefined,
@@ -291,6 +311,7 @@ export function getSegmentsForLayer(
 
   return state.localSegmentationData[layer.name].segments;
 }
+
 export function getVisibleSegments(state: OxalisState): SegmentMap | null | undefined {
   const layer = getVisibleSegmentationLayer(state);
 
@@ -387,9 +408,31 @@ function _getRenderableResolutionForActiveSegmentationTracing(state: OxalisState
 export const getRenderableResolutionForActiveSegmentationTracing = reuseInstanceOnEquality(
   _getRenderableResolutionForActiveSegmentationTracing,
 );
+
 export function getMappingInfoForVolumeTracing(
   state: OxalisState,
   tracingId: string | null | undefined,
 ): ActiveMappingInfo {
   return getMappingInfo(state.temporaryConfiguration.activeMappingByLayer, tracingId);
 }
+
+export function getLastLabelAction(volumeTracing: VolumeTracing): LabelAction | undefined {
+  return volumeTracing.lastLabelActions[0];
+}
+
+export function getLabelActionFromPreviousSlice(
+  state: OxalisState,
+  volumeTracing: VolumeTracing,
+  resolution: Vector3,
+  dim: 0 | 1 | 2,
+): LabelAction | undefined {
+  // Gets the last label action which was performed on a different slice.
+  // Note that in coarser mags (e.g., 8-8-2), the comparison of the coordinates
+  // is done while respecting how the coordinates are clipped due to that resolution.
+  const adapt = (vec: Vector3) => V3.roundElementToResolution(vec, resolution, dim);
+  const position = adapt(getFlooredPosition(state.flycam));
+
+  return volumeTracing.lastLabelActions.find(
+    (el) => Math.floor(adapt(el.centroid)[dim]) != position[dim],
+  );
+}

frontend/javascripts/oxalis/model/actions/volumetracing_actions.ts

@@ -49,9 +49,8 @@ export type ClickSegmentAction = {
   cellId: number;
   somePosition: Vector3;
 };
-export type CopySegmentationLayerAction = {
-  type: "COPY_SEGMENTATION_LAYER";
-  source: "previousLayer" | "nextLayer";
+export type InterpolateSegmentationLayerAction = {
+  type: "INTERPOLATE_SEGMENTATION_LAYER";
 };
 export type MaybeUnmergedBucketLoadedPromise = null | Promise<BucketDataArray>;
 export type AddBucketToUndoAction = {
@@ -118,7 +117,7 @@ export type VolumeTracingAction =
   | FinishAnnotationStrokeAction
   | SetMousePositionAction
   | HideBrushAction
-  | CopySegmentationLayerAction
+  | InterpolateSegmentationLayerAction
   | SetContourTracingModeAction
   | SetSegmentsAction
   | UpdateSegmentAction
@@ -203,9 +202,8 @@ export const updateSegmentAction = (
   layerName,
   timestamp,
 });
-export const copySegmentationLayerAction = (fromNext?: boolean): CopySegmentationLayerAction => ({
-  type: "COPY_SEGMENTATION_LAYER",
-  source: fromNext ? "nextLayer" : "previousLayer",
+export const interpolateSegmentationLayerAction = (): InterpolateSegmentationLayerAction => ({
+  type: "INTERPOLATE_SEGMENTATION_LAYER",
 });
 export const updateDirectionAction = (centroid: Vector3): UpdateDirectionAction => ({
   type: "UPDATE_DIRECTION",

frontend/javascripts/oxalis/model/dimensions.ts

@@ -1,5 +1,5 @@
-import type { OrthoView, Vector3 } from "oxalis/constants";
-import { OrthoViews } from "oxalis/constants";
+import { OrthoView, OrthoViews, Vector3 } from "oxalis/constants";
+
 export type DimensionIndices = 0 | 1 | 2;
 export type DimensionMap = [DimensionIndices, DimensionIndices, DimensionIndices];
 // This is a class with static methods dealing with dimensions and
@@ -69,6 +69,22 @@ const Dimensions = {
     }
   },
 
+  dimensionNameForIndex(dim: DimensionIndices): string {
+    switch (dim) {
+      case 2:
+        return "Z";
+
+      case 0:
+        return "X";
+
+      case 1:
+        return "Y";
+
+      default:
+        throw new Error(`Unrecognized dimension: ${dim}`);
+    }
+  },
+
   roundCoordinate(coordinate: Vector3): Vector3 {
     return [Math.floor(coordinate[0]), Math.floor(coordinate[1]), Math.floor(coordinate[2])];
   },