Merge branch 'master' of github.com:scalableminds/webknossos into cod…

…e_of_conduct * 'master' of github.com:scalableminds/webknossos: fix leowe slack notification (#6239) Volume annotation zarr streaming (#6203) Include segmentation layers in handling maximum number of renderable layers restricted by hardware (#6211) Allow restricting mags for new volume layer (#6229) Added documentation on Connectome Viewer (#6232)
scalableminds · May 25, 2022 · 3c2d09f · 3c2d09f
2 parents 0901d76 + 3f242a5
commit 3c2d09f
Show file tree

Hide file tree

Showing 24 changed files with 567 additions and 174 deletions.
diff --git a/.circleci/slack-notification.sh b/.circleci/slack-notification.sh
@@ -17,7 +17,7 @@ if [ "${CIRCLE_BRANCH}" == "master" ] ; then
     author=${author/valentin-pinkau/<@valentin>}
     author=${author/youri-k/<@youri>}
     author=${author/Dagobert42/<@Arthur>}
-    author=${author/leowe/<@leowe>}
+    author=${author/leowe/<@leo>}
     channel="webknossos-bots"
     commitmsg="$(git log --format=%s -n 1)"
     pullregex="(.*)#([0-9]+)(.*)"

diff --git a/CHANGELOG.unreleased.md b/CHANGELOG.unreleased.md
@@ -15,6 +15,10 @@ For upgrade instructions, please check the [migration guide](MIGRATIONS.released
 - Added a batching mechanism to the task creation via NML to support uploading more than 100 NMLs at a time. [#6216](https://github.com/scalableminds/webknossos/pull/6216)
 - Added a long-running job that applies the merging done via a merger mode tracing to a new output dataset. The job is accessible via a button next to the merger mode button once the merger mode is active. [#6086](https://github.com/scalableminds/webknossos/pull/6086)
 - Added support to stream zarr files using the corresponding [zarr spec](https://zarr.readthedocs.io/en/stable/spec/v2.html#storage). [#6144](https://github.com/scalableminds/webknossos/pull/6144)
+- Added support to stream volume annotations as a zarr data set. [#6203](https://github.com/scalableminds/webknossos/pull/6203)
+- Added segmentation layers to the functionality catching the case that more layers are active that the hardware allows. This prevents rendering issue with more than one segmentation layer and multiple color layers. [#6211](https://github.com/scalableminds/webknossos/pull/6211)
+- Selecting "Download" from the tracing actions now opens a new modal, which lets the user select data for download, start TIFF export jobs or copy code snippets to get started quickly with the webKonossos Python Client. [#6171](https://github.com/scalableminds/webknossos/pull/6171)
+- Adding a New Volume Layer via the left border tab now gives the option to restrict resolutions for the new layer. [#6229](https://github.com/scalableminds/webknossos/pull/6229)
 
 ### Changed
 - When creating a new annotation with a volume layer (without fallback) for a dataset which has an existing segmentation layer, the original segmentation layer is still listed (and viewable) in the left sidebar. Earlier versions simply hid the original segmentation layer. [#6186](https://github.com/scalableminds/webknossos/pull/6186)
@@ -27,7 +31,6 @@ For upgrade instructions, please check the [migration guide](MIGRATIONS.released
 
 ### Fixed
 - Fixed a bug in the task creation, where creation for some tasks with initial volume data would fail. [#6178](https://github.com/scalableminds/webknossos/pull/6178)
-- Fixed a bug in the task cration, where creation for some tasks with initial volume data would fail. [#6178](https://github.com/scalableminds/webknossos/pull/6178)
 - Fixed a rendering bug which could cause an incorrect magnification to be rendered in rare scenarios. [#6029](https://github.com/scalableminds/webknossos/pull/6029)
 - Fixed a bug which could cause a segmentation layer's "ID mapping" dropdown to disappear. [#6215](https://github.com/scalableminds/webknossos/pull/6215)
 

diff --git a/app/controllers/WKRemoteTracingStoreController.scala b/app/controllers/WKRemoteTracingStoreController.scala
@@ -83,4 +83,26 @@ class WKRemoteTracingStoreController @Inject()(
         } yield Ok(Json.toJson(dataSource))
       }
     }
+
+  def dataStoreURIForDataSet(name: String,
+                             key: String,
+                             organizationName: Option[String],
+                             dataSetName: String): Action[AnyContent] =
+    Action.async { implicit request =>
+      tracingStoreService.validateAccess(name, key) { _ =>
+        implicit val ctx: DBAccessContext = GlobalAccessContext
+        for {
+          organizationIdOpt <- Fox.runOptional(organizationName) {
+            organizationDAO.findOneByName(_)(GlobalAccessContext).map(_._id)
+          } ?~> Messages("organization.notFound", organizationName.getOrElse("")) ~> NOT_FOUND
+          organizationId <- Fox.fillOption(organizationIdOpt) {
+            dataSetDAO.getOrganizationForDataSet(dataSetName)(GlobalAccessContext)
+          } ?~> Messages("dataSet.noAccess", dataSetName) ~> FORBIDDEN
+          dataSet <- dataSetDAO.findOneByNameAndOrganization(dataSetName, organizationId) ?~> Messages(
+            "dataSet.noAccess",
+            dataSetName) ~> FORBIDDEN
+          dataStore <- dataSetService.dataStoreFor(dataSet)
+        } yield Ok(Json.toJson(dataStore.url))
+      }
+    }
 }
diff --git a/conf/messages b/conf/messages
@@ -220,6 +220,7 @@ role.added=Added role to {0}
 
 tracing=Annotation
 tracing.notFound=Tracing couldn’t be found
+tracing.wrongMag=Tracing “{0}” does not have mag “{1}”
 
 annotation.create.failed=Failed to create annotation.
 annotation.create.forbidden=You do not have permission to create annotations for this dataset.

diff --git a/conf/webknossos.latest.routes b/conf/webknossos.latest.routes
@@ -39,7 +39,7 @@ PUT           /dataSetConfigurations/default/:organizationName/:dataSetName
 # Users
 POST          /user/tasks/request                                                       controllers.TaskController.request
 GET           /user/tasks/peek                                                          controllers.TaskController.peekNext
-         
+
 GET           /users                                                                    controllers.UserController.list
 GET           /user                                                                     controllers.UserController.current
 GET           /user/tasks                                                               controllers.UserController.tasks(isFinished: Option[Boolean], limit: Option[Int], pageNumber: Option[Int], includeTotalCount: Option[Boolean])
@@ -100,6 +100,7 @@ POST          /tracingstores/:name/handleTracingUpdateReport
 POST          /tracingstores/:name/validateUserAccess                                   controllers.UserTokenController.validateAccessViaTracingstore(name: String, key: String, token: Option[String])
 PUT           /tracingstores/:name                                                      controllers.TracingStoreController.update(name: String)
 GET           /tracingstores/:name/dataSource/:dataSetName                              controllers.WKRemoteTracingStoreController.dataSource(name: String, key: String, organizationName: Option[String], dataSetName: String)
+GET           /tracingstores/:name/dataStoreURI/:dataSetName                            controllers.WKRemoteTracingStoreController.dataStoreURIForDataSet(name: String, key: String, organizationName: Option[String], dataSetName: String)
 
 # User access tokens for datastore authentification
 POST          /userToken/generate                                                       controllers.UserTokenController.generateTokenForDataStore

diff --git a/docs/connectome_viewer.md b/docs/connectome_viewer.md
@@ -0,0 +1,39 @@
+# Synapse and Connectome Viewer
+
+While webKnossos works great for many different imaging environments, the Synapse and Connectome Viewer is designed in particular for Neuroscientists. The Synapse and Connectome viewer offers a way of listing all incoming and outgoing synapses for a given segment/cell in a neuron segmentation for exploration and analysis. 
+
+Synapse locations, type, and partners have to be pre-computed outside of webKnossos, e.g. with [Voxelytics](https://voxelytics.com). webKnossos can load the resulting synaptic connectivity maps, often called Connectome, and offer deeper insights into your dataset.
+
+## Features
+
+- Visualization of all synapse locations for a given segment
+- Visualization of the agglomerate skeleton for a given segment
+- Display of metadata information such as synapse types (e.g., "dendritic shaft", "spine head", "soma"-synapse, etc.)
+- Filtering by synapse type or direction 
+
+![The Synapse and Connectome viewer displaying the incoming and outgoing synpatic connections for a selected neuron.](images/connectome_viewer.jpeg)
+
+## Getting started with the Synapse & Connectome Viewer
+
+The viewer is available from the `Connectome` tab in the right-hand side panel. If your dataset is prepared with a matching connectome file (see below), you can look up the synpases for any segment.
+
+Use the input field at the top of the Connectome Viewer to enter one or several segment IDs. Alternatively, right-click on any segment in the XYZ viewports and select 'Import Agglomerate and Synapses'.
+
+Several segments/cells can be loaded at the same time to highlight their matching synaptic partners.
+
+In addition to loading the synapse locations and visualizing them as with nodes, webKnossos will also load the agglomerate skeleton representation of the selected segment(s) for context.
+
+## Configuration
+For webKnossos to detect and load your Connectome file, you need to place it into a `connectome` sub-directory for a respective segmentation layer, e.g.:
+
+```
+my_dataset                          # Dataset root
+├─ segmentation                     # Dataset layer name (e.g., color, segmentation)
+│  ├─ connectome                    # Parent directory for connectome files
+│  │  ├─ my_connectome.hdf5         # one or more connectome files
+│  │  ├─ different_connectome.hdf5  # one agglomerate file per mapping
+```
+
+
+## Connectome File Format
+The connectome file format is under active development and experiences frequent changes. [Please reach out us for latest file format spec and configuration help](mailto://[email protected]).
diff --git a/docs/data_formats.md b/docs/data_formats.md
@@ -319,20 +319,20 @@ Groups can be freely nested inside each other.
 
 
 ### ID Mapping Files
-webKnossos supports [dynamic, on-demand re-mapping of the segmentation IDs](./volume_annotation.md#mappings--on-demand-agglomeration) allowing you to quickly toggle between different agglomeration strategies for a segmentation layer. These "agglomerate" files need to be pre-computed and put into the correct (sub)-directory inside a segmentation layer for webKnossos to identify and read them (self-hosted instance only).
+webKnossos supports [dynamic, on-demand re-mapping of the segmentation IDs](./volume_annotation.md#mappings--on-demand-agglomeration) allowing you to quickly toggle between different agglomeration strategies for a segmentation layer. These "mapping" files need to be pre-computed and put into the correct (sub)-directory inside a segmentation layer for webKnossos to identify and read them (self-hosted instance only).
 
 webKnossos supports two formats for these agglomerates:
 
 - JSON -> `mappings` directory
 - HDF5 -> `agglomerates` directory
- (JSON-format) or `agglomerates` directory (HDF5-format) :
 
+E.g.:
 ```
-great_dataset                   # Dataset root
-├─ segmentation                 # Dataset layer (^, color, segmentation)
-│  ├─ mappings                  # Magnification step (1, 2, 4, 8, 16 etc.)
-│  │  ├─ my_mapping_file.json   # one or more agglomerate files
-│  │  ├─ different_mapping.json # one agglomerate file per mapping
+my_dataset                      # Dataset root
+├─ segmentation                 # Dataset layer name (e.g., color, segmentation)
+│  ├─ mappings                  # parent directory for all mappings
+│  │  ├─ my_mapping_file.json   # one or more mapping files
+│  │  ├─ different_mapping.json # one mapping file per pre-computed mapping strategy
 ```
 
 

diff --git a/docs/images/connectome_viewer.jpeg b/docs/images/connectome_viewer.jpeg
diff --git a/docs/index.md b/docs/index.md
@@ -13,7 +13,7 @@ webKnossos is also a platform for [showcasing datasets](https://webknossos.org)
 Sign up for a free account on [https://webknossos.org/](https://webknossos.org/) and either upload one of your own datasets, or work with a large selection of published community datasets. 
 
 ## Features
-* Exploration of large 3D image datasets as found in electron-microscopy, synchrotron, CT, MRI, etc.
+* Exploration of large 3D image datasets as found in electron-microscopy, synchrotron, CT, MRI, Micro/Nano-CT
 * Fully browser-based user experience. No installation required
 * Efficient 3D data streaming for quick loading speeds
 * Creation/editing of skeleton (line-segments) and 3D volumetric annotations
@@ -22,9 +22,10 @@ Sign up for a free account on [https://webknossos.org/](https://webknossos.org/)
 * Easy Sharing. Every dataset and annotation can be securely shared as a web link with others
 * Fine-grained access permission and and user roles for secure collaboration
 * [Standalone datastore component](https://github.com/scalableminds/webknossos/tree/master/webknossos-datastore) for flexible deployments
-* [Supported dataset formats: WKW (Optimized), KNOSSOS cubes](./data_formats.md), [Neuroglancer Precomputed, and BossDB](https://github.com/scalableminds/webknossos-connect), and image stacks (converted on upload)
+* [Supported dataset formats: Zarr, WKW (Optimized), KNOSSOS cubes](./data_formats.md), [Neuroglancer Precomputed, and BossDB](https://github.com/scalableminds/webknossos-connect), and image stacks (converted on upload)
 * [Supported image formats](./data_formats.md): Grayscale, Segmentation Maps, RGB, Multi-Channel
 * [3D Mesh Visualization](./mesh_visualization.md)
+* [Integrated Synapse and Connectome Viewer](./connectome_viewer.md)
 * [Documented Python library for API access and integration in custom analysis workflows](https://docs.webknossos.org/webknossos-py/index.html)
 * [Documented frontend API for user scripting](https://webknossos.org/assets/docs/frontend-api/index.html), REST API for backend access
 * Open-source development with [automated test suite](https://circleci.com/gh/scalableminds/webknossos)

diff --git a/docs/terminology.md b/docs/terminology.md
@@ -38,3 +38,11 @@ A **task** specifies a desired annotation result. A **task instance** refers to
 A **project** is a collection of tasks.
 
 See also the [task and projects guide](./tasks.md).
+
+## Segments
+At its lowest-level a **segment** is the collection of several annotated voxels. At a larger level, segments can grow to be the size of whole cell bodies or partial cells, e.g. a single axon.
+
+Typically many segments make up a segmentation. Segments can be painted manually using the webKnossos volume annotation tools or created through third-party programs typically resulting in larger segmentations of a dataset.
+
+## Agglomerates
+An agglomerate is the combination of several (smaller) segments to reconstruct a larger biological structure. Typically an agglomerate combines the fragments of an over-segmentation created by some automated method, e.g. a machine learning system.